Remote-Url: https://www.cis.upenn.edu/~badler/book/SimulatingHumans.pdf
Retrieved-at: 2021-12-31 15:51:15.093375+00:00

SIMULATING HUMANS: COMPUTERGRAPHICS, ANIMATION, AND CONTROLNorman I. BadlerCary B. Phillips1Bonnie L. WebberDepartment of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia, PA 19104-6389Oxford University Pressc(cid:13)1992 Norman I Badler, Cary B. Phillips, Bonnie L. WebberMarch 25, 19991Current address: Paci(cid:12)c Data Images, 1111 Karlstad Dr., Sunnyvale, CA 94089.iTo Ginny, Denise, and MarkiiContents1 Introduction and Historical Background11.1 Why Make Human Figure Models?4: : : : : : : : : : : : : : : :1.2 Historical Roots7: : : : : : : : : : : : : : : : : : : : : : : : : : :1.3 What is Currently Possible?11: : : : : : : : : : : : : : : : : : : :1.3.1 A Human Model must be Structured Like the HumanSkeletal System12: : : : : : : : : : : : : : : : : : : : : : :1.3.2 A Human Model should Move or Respond Like a Human 121.3.3 A Human Model should be Sized According to Permis-sible Human Dimensions14: : : : : : : : : : : : : : : : : :1.3.4 A Human Model should have a Human-Like Appearance 151.3.5 A Human Model must Exist, Work, Act and ReactWithin a 3D Virtual Environment15: : : : : : : : : : : :1.3.6 Use the Computer to Analyze Synthetic Behaviors16: : :1.3.7 An Interactive Software Tool must be Designed for Us-ability18: : : : : : : : : : : : : : : : : : : : : : : : : : : :1.4 Manipulation, Animation, and Simulation19: : : : : : : : : : : :1.5 What Did We Leave Out?20: : : : : : : : : : : : : : : : : : : : :2 Body Modeling232.1 Geometric Body Modeling23: : : : : : : : : : : : : : : : : : : : :2.1.1 Surface and Boundary Models23: : : : : : : : : : : : : : :2.1.2 Volume and CSG Models25: : : : : : : : : : : : : : : : :2.1.3 The Principal Body Models Used27: : : : : : : : : : : : :2.2 Representing Articulated Figures28: : : : : : : : : : : : : : : : :2.2.1 Background29: : : : : : : : : : : : : : : : : : : : : : : : :2.2.2 The Terminology of Peabody30: : : : : : : : : : : : : : :2.2.3 The Peabody Hierarchy31: : : : : : : : : : : : : : : : : :2.2.4 Computing Global Coordinate Transforms33: : : : : : : :2.2.5 Dependent Joints33: : : : : : : : : : : : : : : : : : : : : :2.3 A Flexible Torso Model34: : : : : : : : : : : : : : : : : : : : : :2.3.1 Motion of the Spine36: : : : : : : : : : : : : : : : : : : :2.3.2Input Parameters37: : : : : : : : : : : : : : : : : : : : : :2.3.3 Spine Target Position38: : : : : : : : : : : : : : : : : : :2.3.4 Spine Database38: : : : : : : : : : : : : : : : : : : : : : :iiiivCONTENTS2.4 Shoulder Complex39: : : : : : : : : : : : : : : : : : : : : : : : :2.4.1 Primitive Arm Motions40: : : : : : : : : : : : : : : : : :2.4.2 Allocation of Elevation and Abduction41: : : : : : : : : :2.4.3Implementation of Shoulder Complex41: : : : : : : : : :2.5 Clothing Models45: : : : : : : : : : : : : : : : : : : : : : : : : :2.5.1 Geometric Modeling of Clothes46: : : : : : : : : : : : : :2.5.2 Draping Model48: : : : : : : : : : : : : : : : : : : : : : :2.6 The Anthropometry Database49: : : : : : : : : : : : : : : : : : :2.6.1 Anthropometry Issues49: : : : : : : : : : : : : : : : : : :2.6.2Implementation of Anthropometric Scaling50: : : : : : :2.6.3Joints and Joint Limits51: : : : : : : : : : : : : : : : : :2.6.4 Mass53: : : : : : : : : : : : : : : : : : : : : : : : : : : : :2.6.5 Moment of Inertia53: : : : : : : : : : : : : : : : : : : : :2.6.6 Strength54: : : : : : : : : : : : : : : : : : : : : : : : : : :2.7 The Anthropometry Spreadsheet54: : : : : : : : : : : : : : : : :2.7.1Interactive Access Anthropometric Database56: : : : : :2.7.2 SASS and the Body Hierarchy57: : : : : : : : : : : : : :2.7.3 The Rule System for Segment Scaling57: : : : : : : : : :2.7.4 Figure Creation59: : : : : : : : : : : : : : : : : : : : : : :2.7.5 Figure Scaling59: : : : : : : : : : : : : : : : : : : : : : :2.8 Strength and Torque Display60: : : : : : : : : : : : : : : : : : :2.8.1 Goals of Strength Data Display61: : : : : : : : : : : : : :2.8.2 Design of Strength Data Displays61: : : : : : : : : : : : :3 Spatial Interaction673.1 Direct Manipulation67: : : : : : : : : : : : : : : : : : : : : : : :3.1.1 Translation68: : : : : : : : : : : : : : : : : : : : : : : : :3.1.2 Rotation68: : : : : : : : : : : : : : : : : : : : : : : : : : :3.1.3Integrated Systems69: : : : : : : : : : : : : : : : : : : : :3.1.4 The Jack Direct Manipulation Operator70: : : : : : : : :3.2 Manipulation with Constraints75: : : : : : : : : : : : : : : : : :3.2.1 Postural Control using Constraints75: : : : : : : : : : : :3.2.2 Constraints for Inverse Kinematics77: : : : : : : : : : : :3.2.3 Features of Constraints78: : : : : : : : : : : : : : : : : :3.2.4Inverse Kinematics and the Center of Mass78: : : : : : :3.2.5Interactive Methodology80: : : : : : : : : : : : : : : : : :3.3 Inverse Kinematic Positioning83: : : : : : : : : : : : : : : : : : :3.3.1 Constraints as a Nonlinear Programming Problem86: : :3.3.2 Solving the Nonlinear Programming Problem87: : : : : :3.3.3 Assembling Multiple Constraints91: : : : : : : : : : : : :3.3.4 Sti(cid:11)ness of Individual Degrees of Freedom93: : : : : : : :3.3.5 An Example93: : : : : : : : : : : : : : : : : : : : : : : :3.4 Reachable Spaces94: : : : : : : : : : : : : : : : : : : : : : : : : :3.4.1 Workspace Point Computation Module96: : : : : : : : : :3.4.2 Workspace Visualization97: : : : : : : : : : : : : : : : : :3.4.3 Criteria Selection98: : : : : : : : : : : : : : : : : : : : : :CONTENTSv4 Behavioral Control1014.1 An Interactive System for Postural Control102: : : : : : : : : : :4.1.1 Behavioral Parameters103: : : : : : : : : : : : : : : : : : :4.1.2 Passive Behaviors109: : : : : : : : : : : : : : : : : : : : : :4.1.3 Active Behaviors114: : : : : : : : : : : : : : : : : : : : : :4.2 Interactive Manipulation With Behaviors116: : : : : : : : : : : :4.2.1 The Feet117: : : : : : : : : : : : : : : : : : : : : : : : : :4.2.2 The Center of Mass and Balance117: : : : : : : : : : : : :4.2.3 The Torso120: : : : : : : : : : : : : : : : : : : : : : : : : :4.2.4 The Pelvis123: : : : : : : : : : : : : : : : : : : : : : : : : :4.2.5 The Head and Eyes123: : : : : : : : : : : : : : : : : : : :4.2.6 The Arms123: : : : : : : : : : : : : : : : : : : : : : : : : :4.2.7 The Hands and Grasping126: : : : : : : : : : : : : : : : :4.3 The Animation Interface126: : : : : : : : : : : : : : : : : : : : : :4.4 Human Figure Motions128: : : : : : : : : : : : : : : : : : : : : :4.4.1 Controlling Behaviors Over Time129: : : : : : : : : : : : :4.4.2 The Center of Mass129: : : : : : : : : : : : : : : : : : : :4.4.3 The Pelvis130: : : : : : : : : : : : : : : : : : : : : : : : : :4.4.4 The Torso130: : : : : : : : : : : : : : : : : : : : : : : : : :4.4.5 The Feet130: : : : : : : : : : : : : : : : : : : : : : : : : :4.4.6 Moving the Heels131: : : : : : : : : : : : : : : : : : : : : :4.4.7 The Arms132: : : : : : : : : : : : : : : : : : : : : : : : : :4.4.8 The Hands132: : : : : : : : : : : : : : : : : : : : : : : : :4.5 Virtual Human Control132: : : : : : : : : : : : : : : : : : : : : :5 Simulation with Societies of Behaviors1375.1 Forward Simulation with Behaviors139: : : : : : : : : : : : : : : :5.1.1 The Simulation Model141: : : : : : : : : : : : : : : : : : :5.1.2 The Physical Execution Environment142: : : : : : : : : :5.1.3 Networks of Behaviors and Events144: : : : : : : : : : : :5.1.4Interaction with Other Models145: : : : : : : : : : : : : :5.1.5 The Simulator147: : : : : : : : : : : : : : : : : : : : : : :5.1.6Implemented Behaviors149: : : : : : : : : : : : : : : : : :5.1.7 Simple human motion control150: : : : : : : : : : : : : : :5.2 Locomotion150: : : : : : : : : : : : : : : : : : : : : : : : : : : : :5.2.1 Kinematic Control151: : : : : : : : : : : : : : : : : : : : :5.2.2 Dynamic Control152: : : : : : : : : : : : : : : : : : : : : :5.2.3 Curved Path Walking154: : : : : : : : : : : : : : : : : : :5.2.4 Examples159: : : : : : : : : : : : : : : : : : : : : : : : : :5.3 Strength Guided Motion161: : : : : : : : : : : : : : : : : : : : : :5.3.1 Motion from Dynamics Simulation161: : : : : : : : : : : :5.3.2Incorporating Strength and Comfort into Motion163: : : :5.3.3 Motion Control164: : : : : : : : : : : : : : : : : : : : : : :5.3.4 Motion Strategies167: : : : : : : : : : : : : : : : : : : : : :5.3.5 Selecting the Active Constraints169: : : : : : : : : : : : :5.3.6 Strength Guided Motion Examples170: : : : : : : : : : : :viCONTENTS5.3.7 Evaluation of this Approach173: : : : : : : : : : : : : : : :5.3.8 Performance Graphs173: : : : : : : : : : : : : : : : : : : :5.3.9 Coordinated Motion174: : : : : : : : : : : : : : : : : : : :5.4 Collision-Free Path and Motion Planning180: : : : : : : : : : : :5.4.1 Robotics Background180: : : : : : : : : : : : : : : : : : :5.4.2 Using Cspace Groups181: : : : : : : : : : : : : : : : : : :5.4.3 The Basic Algorithm182: : : : : : : : : : : : : : : : : : : :5.4.4 The Sequential Algorithm183: : : : : : : : : : : : : : : : :5.4.5 The Control Algorithm185: : : : : : : : : : : : : : : : : :5.4.6 The Planar Algorithm186: : : : : : : : : : : : : : : : : : :5.4.7 Resolving Con(cid:13)icts between Di(cid:11)erent Branches186: : : : :5.4.8 Playing Back the Free Path187: : : : : : : : : : : : : : : :5.4.9Incorporating Strength Factors into the Planned Motion 1895.4.10 Examples190: : : : : : : : : : : : : : : : : : : : : : : : : :5.4.11 Completeness and Complexity191: : : : : : : : : : : : : :5.5 Posture Planning192: : : : : : : : : : : : : : : : : : : : : : : : : :5.5.1 Functionally Relevant High-level Control Parameters196: :5.5.2 Motions and Primitive Motions197: : : : : : : : : : : : : :5.5.3 Motion Dependencies197: : : : : : : : : : : : : : : : : : :5.5.4 The Control Structure of Posture Planning199: : : : : : :5.5.5 An Example of Posture Planning200: : : : : : : : : : : : :6 Task-Level Speci(cid:12)cations2076.1 Performing Simple Commands208: : : : : : : : : : : : : : : : : :6.1.1 Task Environment208: : : : : : : : : : : : : : : : : : : : :6.1.2 Linking Language and Motion Generation209: : : : : : : :6.1.3 Specifying Goals209: : : : : : : : : : : : : : : : : : : : : :6.1.4 The Knowledge Base210: : : : : : : : : : : : : : : : : : : :6.1.5 The Geometric Database211: : : : : : : : : : : : : : : : :6.1.6 Creating an Animation211: : : : : : : : : : : : : : : : : :6.1.7 Default Timing Constructs212: : : : : : : : : : : : : : : :6.2 Language Terms for Motion and Space214: : : : : : : : : : : : : :6.2.1 Simple Commands214: : : : : : : : : : : : : : : : : : : : :6.2.2 Representational Formalism215: : : : : : : : : : : : : : : :6.2.3 Sample Verb and Preposition Speci(cid:12)cations217: : : : : : :6.2.4 Processing a sentence219: : : : : : : : : : : : : : : : : : :6.2.5 Summary221: : : : : : : : : : : : : : : : : : : : : : : : : :6.3 Task-Level Simulation222: : : : : : : : : : : : : : : : : : : : : : :6.3.1 Programming Environment223: : : : : : : : : : : : : : : :6.3.2 Task-actions224: : : : : : : : : : : : : : : : : : : : : : : :6.3.3 Motivating Some Task-Actions225: : : : : : : : : : : : : :6.3.4 Domain-speci(cid:12)c task-actions226: : : : : : : : : : : : : : :6.3.5Issues228: : : : : : : : : : : : : : : : : : : : : : : : : : : :6.3.6 Summary231: : : : : : : : : : : : : : : : : : : : : : : : : :6.4 A Model for Instruction Understanding231: : : : : : : : : : : : :CONTENTSvii7 Epilogue2437.1 A Roadmap Toward the Future244: : : : : : : : : : : : : : : : : :7.1.1Interactive Human Models245: : : : : : : : : : : : : : : : :7.1.2 Reasonable Biomechanical Properties245: : : : : : : : : :7.1.3 Human-like Behaviors245: : : : : : : : : : : : : : : : : : :7.1.4 Simulated Humans as Virtual Agents246: : : : : : : : : : :7.1.5 Task Guidance through Instructions246: : : : : : : : : : :7.1.6 Natural Manual Interfaces and Virtual Reality246: : : : :7.1.7 Generating Text, Voice-over, and Spoken Explicationfor Animation247: : : : : : : : : : : : : : : : : : : : : : : :7.1.8 Coordinating Multiple Agents247: : : : : : : : : : : : : : :7.2 Conclusion248: : : : : : : : : : : : : : : : : : : : : : : : : : : : :Bibliography249Index267viiiCONTENTSPrefaceThe decade of the 80’s saw the dramatic expansion of high performancecomputer graphics into domains previously able only to (cid:13)irt with the tech-nology. Among the most dramatic has been the incorporation of real-timeinteractive manipulation and display for human (cid:12)gures. Though actively pur-sued by several research groups, the problem of providing a virtual or synthetichuman for an engineer or designer already accustomed to Computer-Aided De-sign techniques was most comprehensively attacked by the Computer GraphicsResearch Laboratory at the University of Pennsylvania. The breadth of thate(cid:11)ort as well as the details of its methodology and software environment arepresented in this volume.This book is intended for human factors engineers requiring current knowl-edge of how a computer graphics surrogate human can augment their analy-ses of designed environments. It will also help inform design engineers of thestate-of-the-art in human (cid:12)gure modeling, and hence of the human-centereddesign central to the emergent notion of Concurrent Engineering. Finally, itdocuments for the computer graphics community a ma jor research e(cid:11)ort inthe interactive control and motion speci(cid:12)cation of articulated human (cid:12)gures.Many people have contributed to the work described in this book, but thetextual material derives more or less directly from the e(cid:11)orts of our currentand former students and sta(cid:11): Tarek Alameldin, Francisco Azuola, BreckBaldwin, Welton Becket, Wallace Ching, Paul Diefenbach, Barbara Di Eu-ngenio, Je(cid:11)rey Esakov, Christopher Geib, John Granieri, Marc Grosso, Pei-Hwa Ho, Mike Hollick, Moon Jung, Jugal Kalita, Hyeongseok Ko, EunyoungKoh, Jason Koppel, Michael Kwon, Philip Lee, Libby Levison, Gary Monheit,Michael Moore, Ernest Otani, Susanna Wei, Graham Walters, Michael White,Jianmin Zhao, and Xinmin Zhao. Additional animation help has come fromLeanne Hwang, David Haynes, and Brian Stokes. John Granieri and MikeHollick helped considerably with the photographs and (cid:12)gures.This work would not have been possible without the generous and oftenlong term support of many organizations and individuals. In particular wewould like to acknowledge our many colleagues and friends: Barbara Woolford,Geri Brown, Jim Maida, Abhilash Pandya and the late Linda Orr in the CrewStation Design Section and Mike Greenisen at NASA Johnson Space Center;Ben Cummings, Brenda Thein, Bernie Corona, and Rick Kozycki of the U.S.Army Human Engineering Laboratory at Aberdeen Proving Grounds; JamesHartzell, James Larimer, Barry Smith, Mike Prevost, and Chris Neukom ofthePro ject in the Aero(cid:13)ight Dynamics Directorate of NASA Ames Re-3AIsearch Center; Steve Paquette of the U. S. Army Natick Laboratory; JagdishChandra and David Hislop of the U. S. Army Research O(cid:14)ce; the Army Arti-(cid:12)cial Intelligence Center of Excellence at the University of Pennsylvania andits Director, Aravind Joshi; Art Iverson and Jack Jones of the U.S. ArmyTACOM; Jill Easterly, Ed Boyle, John Ianni, and Wendy Campbell of theU. S. Air Force Human Resources Directorate at Wright-Patterson Air ForceBase; Medhat Korna and Ron Dierker of Systems Exploration, Inc.; Pete GlorCONTENTSixand Joseph Spann of Hughes Missile Systems (formerly General Dynamics,Convair Division); Ruth Maulucci of MOCO Inc.; John McConville, BruceBradtmiller, and Bob Beecher of Anthropology Research Pro ject, Inc.; Ed-mund Khouri of Lockheed Engineering and Management Services; Barb Fechtof Battelle Paci(cid:12)c Northwest Laboratories; Jerry Duncan of Deere and Com-pany; Ed Bellandi of FMC Corp.; Steve Gulasy of Martin-Marietta DenverAerospace; Joachim Grollman of Siemens Research; Kathleen Robinette of theArmstrong Medical Research Lab at Wright-Patterson Air Force Base; HarryFrisch of NASA Goddard Space Flight Center; Jerry Allen and the folks at Sil-icon Graphics, Inc.; Jack Scully of Ascension Technology Corp.; the NationalScience Foundation CISE Grant CDA88-22719 and ILI Grant USE-9152503;and the State of Pennsylvania Benjamin Franklin Partnership. Martin Zaidelcontributed valuable LTX help. Finally, the encouragement and patience ofaEDon Jackson at Oxford University Press has been most appreciated.Norman I. BadlerUniversity of PennsylvaniaCary B. PhillipsPDI, SunnyvaleBonnie L. WebberUniversity of PennsylvaniaxCONTENTSChapter 1Introduction andHistorical BackgroundPeople are all around us. They inhabit our home, workplace, entertainment,and environment. Their presence and actions are noted or ignored, enjoyed ordisdained, analyzed or prescribed. The very ubiquitousness of other people inour lives poses a tantalizing challenge to the computational modeler: peopleare at once the most common ob ject of interest and yet the most structurallycomplex. Their everyday movements are amazingly (cid:13)uid yet demanding toreproduce, with actions driven not just mechanically by muscles and bonesbut also cognitively by beliefs and intentions. Our motor systems manageto learn how to make us move without leaving us the burden or pleasureof knowing how we did it. Likewise we learn how to describe the actionsand behaviors of others without consciously struggling with the processes ofperception, recognition, and language.A famous Computer Scientist, Alan Turing, once proposed a test to deter-mine if a computational agent is intelligent [Tur63]. In the Turing Test, a sub-ject communicates with two agents, one human and one computer, througha keyboard which e(cid:11)ectively restricts interaction to language. The sub jectattempts to determine which agent is which by posing questions to both ofthem and guessing their identities based on the \intelligence" of their answers.No physical manifestation or image of either agent is allowed as the processseeks to establish abstract \intellectual behavior," thinking, and reasoning.Although the Turing Test has stood as the basis for computational intelli-gence since 1963, it clearly omits any potential to evaluate physical actions,behavior, or appearance.Later, Edward Feigenbaum proposed a generalized de(cid:12)nition that includedaction: \Intelligent action is an act or decision that is goal-oriented, arrivedat by an understandable chain of symbolic analysis and reasoning steps, andis one in which knowledge of the world informs and guides the reasoning."[Bod77]. We can imagine an analogous \Turing Test" that would have the12CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDsub ject watching the behaviors of two agents, one human and one synthetic,while trying to determine at a better than chance level which is which. Humanmovement enjoys a universality and complexity that would de(cid:12)nitely challengean animated (cid:12)gure in this test: if a computer-synthesized (cid:12)gure looks, moves,and acts like a real person, are we going to believe that it is real? On the sur-face the question almost seems silly, since we would rather not allow ourselvesto be fooled. In fact, however, the question is moot though the premises areslightly di(cid:11)erent: cartoon characters are hardly \real," yet we watch them andproperly interpret their actions and motions in the evolving context of a story.Moreover, they are not \realistic" in the physical sense { no one expects tosee a manifest Mickey Mouse walking down the street. Nor do cartoons evenmove like people { they squash and stretch and perform all sorts of actionsthat we would never want to do. But somehow our perceptions often makethese characters: they appear to act in a goal-directed way becausebelievabletheir human animators have imbued them with physical \intelligence" andbehaviors that apparently cause them to chase enemies, bounce o(cid:11) walls, andtalk to one another. Of course, these ends are achieved by the skillful weavingof a story into the crafted images of a character. Perhaps surprisingly, themechanisms by which motion, behavior, and emotion are encoded into car-toons isby building synthetic models of little creatures with muscles andnotnerves. The requisite animator skills do not come easily; even in the cartoonworld re(cid:12)nements to the art and technique took much work, time, and study[TJ81]. Creating such movements automatically in response to real-time in-teractive queries posed by the sub ject in our hypothetical experiment does notmake the problem any easier. Even Turing, however, admitted that the intel-ligence sought in his original test did not require the computationalprocessof thinking to be identical to that of the human: the external manifestationin a plausible and reasonable answer was all that mattered.So why are we willing to assimilate the truly arti(cid:12)cial reality of cartoons {characters created and moved entirely unlike \real" people { yet be skepticalof more human-like forms? This question holds the key to our physical TuringTest: as the appearance of a character becomes more human, our perceptualapparatus demands motion qualities and behaviors which sympathize withour expectations. As a cartoon character takes on a human form, the onlycurrently viable method for accurate motion is the recording of a real actorand the tracing or transfer (\rotoscoping") of that motion into the animation.Needless to say, this is not particularly satisfying to the modeler: the motionand actor must exist prior to the synthesized result. Even if we recordedthousands of individual motions and retrieved them through some kind ofindexed video, we would still lack the freshness, variability, and adaptabilityof humans to live, work, and play in an in(cid:12)nite variety of settings.If synthetic human motion is to be produced without the bene(cid:12)t of prior\real" execution and still have a shot at passing the physical Turing Test, thenmodels must carefully balance structure, shape, and motion in a compatiblepackage. If the models are highly simpli(cid:12)ed or stylized, cartoons or caricatureswill be the dominant perception; if they look like humans, then they will be3expected to behave like them. How to accomplish this without a real actorshowing the way is the challenge addressed here.Present technology can approach human appearance and motion throughcomputer graphics modeling and three-dimensional animation, but there isconsiderable distance to go before purely synthesized (cid:12)gures trick our senses.A number of promising research routes can be explored and many are tak-ing us a considerable way toward that ultimate goal. By properly delimitingthe scope and application of human models, we can move forward, not to re-place humans, but to substitute adequate computational surrogates in varioussituations otherwise unsafe, impossible, or too expensive for the real thing.The goals we set in this study are realistic but no less ambitious than thephysical Turing Test: we seek to build computational models of human-like(cid:12)gures which, though they may not trick our senses into believing they arealive, nonetheless manifest animacy and convincing behavior. Towards thisend, we(cid:15)Create an interactive computer graphics human model.(cid:15)Endow it with reasonable biomechanical properties.(cid:15)Provide it with \human-like" behaviors.(cid:15)Use this simulated (cid:12)gure as an agent to e(cid:11)ect changes in its world.(cid:15)Describe and guide its tasks through natural language instructions.There are presently no perfect solutions to any of these problems, but sig-ni(cid:12)cant advances have enabled the consideration of the suite of goals underuniform and consistent assumptions. Ultimately, we should be able to giveour surrogate human directions that, in conjunction with suitable symbolicreasoning processes, make it appear to behave in a natural, appropriate, andintelligent fashion. Compromises will be essential, due to limits in computa-tion, throughput of display hardware, and demands of real-time interaction,but our algorithms aim to balance the physical device constraints with care-fully crafted models, general solutions, and thoughtful organization.This study will tend to focus on one particularly well-motivated applicationfor human models: human factors analysis. While not as exciting as motionpicture characters, as personable as cartoons, or as skilled as Olympic athletes,there are justi(cid:12)able uses to virtual human (cid:12)gures in this domain. Visualizingthe appearance, capabilities and performance of humans is an important anddemanding application (Plate 1). The lessons learned may be transferred toless critical and more entertaining uses of human-like models. From modelingrealistic or at least reasonable body size and shape, through the control ofthe highly redundant body skeleton, to the simulation of plausible motions,human (cid:12)gures o(cid:11)er numerous computational problems and constraints. Build-ing software for human factors applications serves a widespread, non-animatoruser population. In fact, it appears that such software has broader applica-tion since the features needed for analytic applications { such as multiple4CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDsimultaneous constraints { provide extremely useful features for the conven-tional animator. Our software design has tried to take into account a widevariety of physical problem-oriented tasks, rather than just o(cid:11)er a computergraphics and animation tool for the already skilled or computer-sophisticatedanimator.The remainder of this chapter motivates the human factors environmentand then traces some of the relevant history behind the simulation of human(cid:12)gures in this and other domains. It concludes with a discussion of the speci(cid:12)cfeatures a human modeling and animation system should have and why wehave concentrated on some and not others. In particular, we are not consid-ering cognitive problems such as perception or sensory interpretation, targettracking, ob ject identi(cid:12)cation, or control feedback that might be importantparts of some human factors analyses. Instead we concentrate on modeling avirtual human with reasonable biomechanical structure and form, as describedin Chapter 2. In Chapter 4 we address the psychomotor behaviors manifestedby such a (cid:12)gure and show how these behaviors may be interactively accessedand controlled. Chapter 5 presents several methods of motion control thatbridge the gap between biomechanical capabilities and higher level tasks. Fi-nally, in Chapter 6 we investigate the cognition requirements and strategiesneeded to have one of these computational agents follow natural language taskinstructions.1.1 Why Make Human Figure Models?Our research has focused on software to make the manipulation of a simulatedhuman (cid:12)gure easy for a particular user population: human factors design en-gineers or ergonomics analysts. These people typically study, analyze, assess,and visualize human motor performance, (cid:12)t, reach, view, and other physicaltasks in a workplace environment. Traditionally, human factors engineers an-alyze the design of a prototype workplace by building a mock-up, using realsub jects to perform sample tasks, and reporting observations about designsatisfaction. This is limiting for several reasons. Jerry Duncan, a human fac-tors engineer at Deere & Company, says that once a design has progressedto the stage at which there is su(cid:14)cient information for a model builder toconstruct the mock-up, there is usually so much inertia to the design thatradical changes are di(cid:14)cult to incorporate due to cost and time considera-tions. After a design goes into production, de(cid:12)ciencies are alleviated throughspecialized training, limits on physical characteristics of personnel, or vari-ous operator aids such as mirrors, markers, warning labels, etc. The goal ofcomputer-simulated human factors analysis is not to replace the mock-up pro-cess altogether, but to incorporate the analysis into early design stages so thatdesigners can eliminate a high proportion of (cid:12)t and function problems beforebuilding the mock-ups. Considering human factors and other engineering andfunctional analyses together during rather than after the ma jor design processis a hallmark of Concurrent Engineering [Hau89].1.1. WHY MAKE HUMAN FIGURE MODELS?5It is di(cid:14)cult to precisely characterize the types of problems a human fac-tors engineer might address. Diverse situations demand empirical data onhuman capabilities and performance in generic as well as highly speci(cid:12)c tasks.Here are some examples.(cid:15)Population studies can determine body sizes representative of somegroup, say NASA astronaut trainees, and this information can be usedto determine if space vehicle work cells are adequately designed to (cid:12)tthe individuals expected to work there. Will all astronauts be able to(cid:12)t through doors or hatches? How will changes in the workplace designa(cid:11)ect the (cid:12)t? Will there be unexpected obstructions to zero gravitylocomotion? Where should foot- and hand-holds be located?(cid:15)An individual operating a vehicle such as a tractor will need to seethe surrounding space to execute the task, avoid any obstructions, andinsure safety of nearby people. What can the operator see from a par-ticular vantage point? Can he control the vehicle while looking out therear window? Can he see the blade in order to follow an excavation line?(cid:15)Speci(cid:12)c lifting studies might be performed to determine back strainlimits for a typical worker population. Is there room to perform a liftproperly? What joints are receiving the most strain? Is there a betterposture to minimize torques? How does placement of the weight andtarget a(cid:11)ect performance? Is the worker going to su(cid:11)er fatigue after afew iterations?(cid:15)Even more specialized experiments may be undertaken to evaluate thecomfort and feel of a particular tool’s hand grip. Is there su(cid:14)cient roomfor a large hand? Is the grip too large for a small hand? Are all thecontrols reachable during the grip?The answers to these and other questions will either verify that the designis adequate or point to possible changes and improvements early in the designprocess. But once again, the diversity of human body sizes coupled withthe multiplier of human action and interaction with a myriad things in theenvironment leads to an explosion in possible situations, data, and tests.Any desire to build a \complete" model of human behavior, even for thehuman factors domain, is surely a futile e(cid:11)ort. The (cid:12)eld is too broad, theliterature immense, and the theory largely empirical. There appear to betwo directions out of this dilemma. The (cid:12)rst would be the construction ofa computational database of all the known, or at least useful, data. Vari-ous e(cid:11)orts have been undertaken to assemble such material, for example, theNASA sourcebooks [NAS78, NAS87] and theEngineering Data Compendium[BKT86, BL88]. The other way is to build a sophisticated computationalhuman model and use it as a sub ject in simulated virtual environment tests.The model will utilize an ever-expanding human factors data set to dictate itsperformance. Upon some re(cid:13)ection, it appears that database direction may6CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDstart out as the smoother road, but it quickly divides into numerous sinu-ous paths pot-holed with data gaps, empirical data collection limitations, andpopulation-speci(cid:12)c dependencies. The alternative direction (using a compu-tational model underlying any data) may be harder to construct at (cid:12)rst, andmay have many detours for awhile, but gradually it leads to more destinationswith better roads.This metaphor carries a philosophy for animating human movement thatderives from a computer science rather than an empirical point of view. Wecannot do without the e(cid:11)orts of the human factors community, but we cannotuse their workas the starting point for human (cid:12)gure modeling. Com-per seputer scientists seek computationally general yet e(cid:14)cient solutions to prob-lems. Human factors engineers often analyze a succession of speci(cid:12)c tasksor situations. The role we play is transforming the speci(cid:12)c needs of the en-gineer or analyst into a generalized setting where some large percentage ofsituations may be successfully analyzed. There is su(cid:14)cient research requiredto solve general yet di(cid:14)cult problems to justify building suitable software ina computer science environment. The expectation is that in the long runa more speci(cid:12)c case-by-case implementation approach will be economicallyimpractical or technologically infeasible.As we continue to interact with human factors specialists, we have come toappreciate the broad range of problems they must address: (cid:12)t, reach, visibility,comfort, access, strength, endurance, and fatigue, to mention only some oftheones. Our approach is not a denial of their perception andnon-cognitiveanalysismodeling, rather it is an alternative view of the problem as. Broadlyspeaking, modeling is the embodiment within computer databases or programsof worldly phenomena. Models can be of many types:(cid:15)Mathematical formulations: physical equations of motion, limb strengthin tables of empirical data, evaluation formulas measuring workload orfatigue.(cid:15)Geometric and topological models: structures representing workplaceob jects, human body segments, paths to follow, joints and joint limits,spaces that can be reached, attachments, and constraints.(cid:15)Conceptual models: names for things, attributes such as color, (cid:13)exibil-ity, and material, relationships between ob jects, functional properties.Of course, modeling (especially of the (cid:12)rst sort) is a signi(cid:12)cant and fun-damental part of many studies in the human factors domain, but it has beendi(cid:14)cult to balance the needs of the engineer against the complexity of themodeling software. Often, the model is elaborated in only a few dimensionsto study some problem while no global integration of models is attempted.Clearly the broadest interpretation of modeling draws not only from manyareas of computer science such as arti(cid:12)cial intelligence, computer graphics,simulation, and robotics, but also from the inherently relevant (cid:12)elds of biome-chanics, anthropometry, physiology, and ergonomics.1.2. HISTORICAL ROOTS7The challenge to embed a reasonable set of capabilities in an integratedsystem has provided dramatic incentives to study issues and solutions in three-dimensional interaction methodologies, multiple goal positioning, visual (cid:12)eldassessment, reach space generation, and strength guided motion, to name afew. The empirical data behind these processes is either determined fromreliable published reports or supplied by the system user. By leaving theactual data open to the user, the results are as valid as the user wishes tobelieve. While this is not a very pleasant situation, the inherent variabilityin human capability data makes some error unavoidable. Better, we think,to let the user know or control the source data than to hide it. This attitudetoward validation is not the only plausible one, but it does permit (cid:13)exibilityand generality for the computer and allows (cid:12)nal judgment to be vested in theuser.Lest there be concern that we have pared the problem down so far thatlittle of interest remains, we change tactics for awhile and present an historicalview of e(cid:11)orts to model humans and their movements. By doing so, we shoulddemonstrate that the human factors domain mirrors problems which arise inother contexts such as dance, sports, or gestural communication. The criteriafor success in these (cid:12)elds may be more stringent, so understanding the roleand scope of human movement in them can only serve to strengthen ourunderstanding of more mundane actions.1.2 Historical RootsInteractive computer graphics systems to support human (cid:12)gure modeling,manipulation, and animation have existed since the early seventies. We tracerelevant developments with a sense more of history and evolution rather thanof exhaustive survey. There are numerous side branches that lead to interest-ing topics, but we will sketch only a few of those here.Three-dimensional human (cid:12)gure models apparently arose independentlyfrom at least six di(cid:11)erent applications.1. Crash simulation. Automobile and aircraft safety issues led to the de-velopment of sophisticated codes for linked mass deceleration studies.These programs generally ran in batch mode for long hours on main-frame computers. The results were tabulated, and in some cases con-verted to a form that could animate a simple 3D mannequin model[Fet82, Wil82, BOT79]. The application was characterized by non-interactive positioning and force-based motion computations with anal-ysis of impact forces to a(cid:11)ected body regions and subsequent injuryassessment.2. Motion analysis. Athletes, patients with psychomotor disabilities, ac-tors, or animals were photographed in motion by one or more (cid:12)xed,calibrated cameras. The two-dimensional information was correlatedbetween views and reconstructed as timed 3D data points [RA90]. This8CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDdata could be (cid:12)ltered and di(cid:11)erentiated to compute velocities, accelera-tions, torques and forces. Visualization of the original data validated thedata collection process, but required human (cid:12)gure models. Often justwire-frames, they served in a support role for athletic performance im-provement, biomechanical analysis, cartoon motion [TJ81], and trainingor physical therapy [Win90]. Related e(cid:11)orts substituted direct or activesensing devices for photographic processing [CCP80]. Presently, activemotion sensing is used not only for performance analysis but gesturalinput for virtual environments (for example, [FMHR87, BBH90] and+many others).3. Workplace assessment. The earliest system with widespread use wasSAMMIE [KSC81]. This problem domain is characterized by interac-tive body positioning requirements and analyses based on visual inspec-tion of 3D computer graphics models. Fast interaction with wire-framedisplays provided dynamic feedback to the workplace evaluator. Othermodeling tools were developed, such as CAR II [HBD80], Combiman++[BEK81], and Crew Chief [MKK88, EI91] to provide validated an-thropometric or capability data for real populations.4. Dance or movement notation. The speci(cid:12)cation of self-generated, pur-posive, aesthetically-pleasing, human movement has been the sub jectof numerous notational systems [Hut84]. Dance notations were consid-ered as a viable, compact, computationally tractable mode of expres-sion for human movement due to their re(cid:12)nement as symbolic motiondescriptions [BS79]. An animation was to be the debugging tool to val-idate the correctness of a given notated score. The direct creation ofmovement through a notational or numeric interface was also considered[CCP80, CCP82, HE78].5. Entertainment. People (or at least animate creatures) are the favoritesub ject of cartoons and movies. Two-dimensional animation techniqueswere the most widely used [Cat72, BW76, Lev77, Cat78].In an ef-fort to avoid rotoscoping live actors, early 3D modeling and animationtechniques were developed at the University of Utah, Ohio State Univer-sity, and the New York Institute of Technology [Wes73, Hac77, Stu84,Gom84, HS85a].6. Motion understanding. There are deep connections between humanmotion and natural language. One of these attempted to produce asort of narration of observed (synthetic) movement by characterizingchanges in spatial location or orientation descriptions over time [Bad75].More recently, the inverse direction has been more challenging, namely,producing motion from natural language descriptions or instructions[BWKE91, TST87].Our earliest e(cid:11)orts were directed at language descriptions of ob ject motion.Speci(cid:12)cally, we created representations of 3D ob ject motion and directional1.2. HISTORICAL ROOTS9adverbials in such a way that image sequences of moving ob jects could beanalyzed to produce English sentence motion descriptions [Bad75, Bad76]. Wethen extended the model to articulated (cid:12)gures, concentrating on graphicallyvalid human (cid:12)gure models to aid the image understanding process [BOT79].This e(cid:11)ort led to the work of Joseph O’Rourke, who attempted model-drivenanalysis of human motion [OB80] using novel 3D constraint propagation andgoal-directed image understanding.To improve the motion understanding component we focused on motionrepresentations specially designed for human movement [WSB78, BS79]. Anin-depth study of several human movement notation systems (such as La-banotation [Hut70] and Eshkol-Wachmann [Hut84]) fostered our appreciationfor the breadth and complexity of human activities. Our early attempts tore-formulate Labanotation in computational models re(cid:13)ected a need to coverat least the space of human (skeletal) motion. We investigated input sys-tems for Labanotation [BS76, Hir77], although later they were discarded as agenerally accessible means of conveying human movement information fromanimator to computer (cid:12)gure: there was simply too much overhead in learningthe nuances and symbology of the notational system. Moreover, concurrentdevelopments in three-dimensional interactive computer graphics o(cid:11)ered morenatural position and motion speci(cid:12)cation alternatives. The (cid:12)nal blow to us-ing Labanotation was its lack of dynamic information other than timing andcrude \accent" and phrasing marks.The movement representations that we developed from Labanotation, how-ever, retained one critically important feature: goal-directedness for e(cid:14)cientmotion speci(cid:12)cation. Given goals, processes had to be developed to satisfythem. A simulation paradigm was adopted and some of the special problemsof human movement simulation were investigated [BSOW78, BOK80, KB82].Others studied locomotion [CCP82, Zel82, GM85, Gir87, Bru88, BC89], whilewe concentrated on inverse kinematics for reach goals [KB82, Kor85]. We wereespecially anxious to manage multiple reach and motion goals that mutuallya(cid:11)ected many parts of the body. Solving this problem in particular led to laterre-examination of constraint-based positioning and more general and robustalgorithms to achieve multiple simultaneous goals [BMW87, ZB89]. Chapter4 will discuss our current approach in detail.Our study of movement notations also led to an appreciation of certainfundamental limitations most of them possessed: they were good at describingthe changes or end results of a movement (should be done), but werewhatcoarse or even non-speci(cid:12)c when it came to indicatinga movement oughthowto be performed. The notator’s justi(cid:12)cation was that the performer (for ex-ample, a dancer) was anwho knew from experience and trainingexpert systemjust how to do the notated motion. The transformation from notation intosmooth, natural, expressive movements was part of the art. The exceptionto the nearly universal failure of notational systems to capture nuances ofbehavior was E(cid:11)ort-Shape notation [Del70, BwDL80]. We began a study ofpossible computational analogues to the purely descriptive semantics of thatsystem. By 1986 a model of human movement emerged which integrated the10CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDkinematic and inverse kinematic approach with a dynamic, force-based model[Bad89]. Ma jor contributions to dynamics-based animation were made byothers, notably [AG85, AGL87, WB85, Wil86, Wil87, IC87, HH87, Hah88].Recently we combined some of the characteristics of the dynamics approach{ the use of physical torques at the body joints { with goal-directed behaviorto achieve([LWZB90] and Chapter 5).strength guided motionWhile we were actively engaged in the study of motion representations,concurrent developments in interactive systems for the graphical manipula-tion of a computerized (cid:12)gure were being actively implemented at the Uni-versity of Pennsylvania. Implementation and development has been a strongexperimental component of our research, from the early positioning languagebased on Labanotation concepts [WSB78], to the next generation frame bu(cid:11)er-based system called[Kor85, BKK85], to the direct manipulationTEMPUSof the (cid:12)gure with a 6-axis digitizer [BMB86], and (cid:12)nally to our present SiliconGraphics workstation-based system[PB88, PZB90] (Chapters 2 and+TM1Jack4).As our experience with interactive graphical manipulation of a (cid:12)gure ma-tured, we returned to the connections between language and motion we hadbegun in the mid-1970’s. The manipulation of the (cid:12)gure for task analysisbegged for more e(cid:14)cient means of specifying the task. So we began to in-vestigate natural language control for task animation [Gan85, BG86, Kar87,Kar88, Kal90, KB90, KB91]. New representations for motion verbs and tech-niques for de(cid:12)ning and especiallytheir semantics were investigated.executingA simple domain of panel-type ob jects and their motions were studied by Je(cid:11)Gangel. Robin Karlin extended the semantics to certain temporal adverbials(such as repetitions and culminations) in a domain of kitchen-ob jects. Ourpresent e(cid:11)ort is exempli(cid:12)ed here by the work of Jugal Kalita and Libby Lev-ison. Kalita studied verbs of physical manipulation and used constraints in afundamental fashion to determine generalized verb semantics. Levison makesexplicit connections between a verb’s semantic representation and the sorts ofprimitive behaviors and constraints known to be directly simulatable by theJackanimation system (Chapter 6).Given that natural language or some other arti(cid:12)cial language was to beused to describe tasks or processes, a suitable simulation methodology hadto be adopted.In his HIRES system, Paul Fishwick investigated task andprocess simulation for human animation [Fis86, Fis88]. Output was producedby selecting from among pre-de(cid:12)ned key postures. For example, an animationof the famous \Dining Philosophers" problem using (cid:12)ve human (cid:12)gure modelswas produced by simulation of the petri net solution in the HIRES simula-tor. By 1989 we replaced HIRES by a new simulation system, YAPS, whichincorporated temporal planning with imprecise speci(cid:12)cations [KKB88], taskinterruption, and task time estimation based on human performance models[EBJ89, EB90, BWKE91] (Chapter 6).This brings us to the presentsystem structure designed to accommo-Jack1Jackis a registered trademark of the University of Pennsylvania.1.3. WHAT IS CURRENTLY POSSIBLE?11date as many of the historical applications as possible within an integratedand consistent software foundation. To begin to describe that, we need toreview what human modeling capabilities are needed and what problem im-plementation choices we might make.1.3 What is Currently Possible?Since we wish to animate synthetic human (cid:12)gures primarily in the humanfactors engineering domain, we should decide what features are essential, de-sirable, optional, or unnecessary. Only by prioritizing the e(cid:11)ort can such alarge-scale undertaking be managed. Given priorities, implementation tech-niques and trade-o(cid:11)s may be investigated. Though we may often draw on ex-isting knowledge and algorithms, there are many fundamental features whichwe may have to invent or evolve due to the speci(cid:12)c structure of the human(cid:12)gure, characteristics of human behavior, timing demands of real-time in-teraction, or limitations of the display hardware. Accordingly, a variety ofhuman (cid:12)gure modeling issues will be examined here to introduce and justifythe choices we have made in our broad yet integrated e(cid:11)ort.The embodiment of our choices for human modeling is a software systemcalled. Designed to run on Silicon Graphics 4D workstations,isJackJackused for the de(cid:12)nition, manipulation, animation, and human factors perfor-mance analysis of virtual human (cid:12)gures. Built on a powerful representationfor articulated (cid:12)gures,o(cid:11)ers the interactive user a simple, intuitive, andJackyet extremely capable interface into any three dimensional world.incor-Jackporates sophisticated yet highly usable algorithms for anthropometric human(cid:12)gure generation, a (cid:13)exible torso, multiple limb positioning under constraints,view assessment, reach space generation, and strength guided performancesimulation of human (cid:12)gures. Of particular importance is a simulation levelwhich allows access toby high level task control, various knowledge bases,Jacktask de(cid:12)nitions and natural language instructions. Thus human activities canbe visualized from high level task understanding and planning as well as byinteractive speci(cid:12)cation.One can think ofas an experimental environment in which a numberJackof useful general variables may be readily created, adjusted, or controlled: theworkplace, the task, the human agent(s) and some responses of the workplaceto internally or externally controlled actions. The results of speci(cid:12)c instancesof these input parameters are reported through computer graphics displays,textual information, and animations. Thus the (cid:12)eld of view of a 50percentilethmale while leaning over backwards in a tractor seat as far a possible may bedirectly visualized through the graphic display. If the (cid:12)gure is supposed towatch the corner of the bulldozer blade as it moves through its allowed motion,the human (cid:12)gure’s gaze will follow in direct animation of the view.In the following subsections, several desiderata are presented for humanmodels. Under each, we summarize the ma jor features { with justi(cid:12)cationsand bene(cid:12)ts { of thesoftware. The detailed discussions of these featuresJack12CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDand their implementation constitute the bulk of the remaining chapters.1.3.1 A Human Model must be Structured Like the Hu-man Skeletal SystemTo build a biomechanically reasonable (cid:12)gure, the skeletal structure shouldresemble but need not copy that of humans. We can bu(cid:11)er the complexityof actual bone shapes, joint types and joint contact surfaces with require-ments for interactive use and external motion approximations. For example,rotational joints are usually assumed to have a virtual center about whichthe adjacent body segments move. While such simpli(cid:12)cations would not beappropriate for, say, knee prosthesis design, there appears to be little harmin variations on the order of a centimeter or so. Of course, there are situ-ations where small departures from reality could a(cid:11)ect the verisimilitude ofthe (cid:12)gure; accordingly we have concentrated on rather accurate models forthe torso and shoulder complex. Many other software systems (or manualmethods) presume a (cid:12)xed shoulder joint but it is obvious that this is not trueas the arm is elevated.1.has a fully linked body model including a 17 segment (cid:13)exible torsoJackwith vertebral joint limits. In general, individual joints may have one,two, or three degrees of freedom (DOFs). Related groups of joints,such as the spine or the shoulder complex, may be manipulated as aunit. The result is reasonable biomechanical realism with only modestcomputational overhead.2. Theshoulder mass joint center is posture-dependent. AccurateJackshoulder motion is modeled through an explicit dependency betweenarm position and clavicle rotation. The (cid:12)gure therefore presents appro-priate shoulder and clavicle motions during positioning. The shoulderjoint has spherical (globographic [EP87]) limits for improved motionrange accuracy.3. All joint rotations are sub ject to limits. During manipulation, rotationspropagate when joint limits would be exceeded.4. A fully articulated hand model is attached.5. The foot is articulated enough to provide toe and heel (cid:13)exibility. If moreDOFs were required, they could be easily added.1.3.2 A Human Model should Move or Respond Like aHumanIdeally, the motions presented by a simulated (cid:12)gure will be biomechanicallyvalid. They should not only appear \human-like," but they should be vali-dated against empirical data for real sub jects under similar conditions. This1.3. WHAT IS CURRENTLY POSSIBLE?13goal is desirable but di(cid:14)cult to reach in a generalized motion model preciselybecause such a model must allow interpolation and extrapolation to situationsother than those originally measured. Models are needed to provide reason-able interpretations of data that by necessity must be sampled rather coarsely,in speci(cid:12)c situations, and with a collection of speci(cid:12)c sub jects. The closest wecan get to the ideal is to provide generic mechanisms that incorporate when-ever possible empirical data that a user believes to be valid up to the degreeof error permitted for the task. Rather than hide such data,takes theJackview of an open database where reasonable default human anthropometric,strength or performance data is provided, but user customizing is the rulerather than the exception.1.permits multiple (cid:12)gures to simultaneously inhabit an environment.JackMulti-person environments and operator interactions may be studied forinterference, view, and coordinated tasks.2. A number of active behaviors are de(cid:12)ned for the (cid:12)gure and may be se-lected or disabled by the user. Among the most interesting is constrain-ing the center of mass of the entire (cid:12)gure during manipulation. Thisallows automatic balance and weight-shifting while other tasks such asreaching, viewing, bending over, etc. are being performed. Spare cycleson the workstation are used inbetween human operator inputs to con-stantly monitor the active constraints and move the body joints towardstheir satisfaction.3. People usually move in ways that conserve resources (except when theyare deliberately trying to achieve optimum performance). If strengthinformation as a function of body posture and joint position is available,that data may be used to predict certain end-e(cid:11)ector motion paths underspeci(cid:12)ed \comfort" conditions. Thus the exact motions involved in,say, lifting a weight are subservient to the strength model, comfort andfatigue parameters, and heuristics for selecting among various movementstrategies. By avoiding \canned" or arbitrary (for example, straightline) motion paths, great (cid:13)exibility in executing tasks is provided.4. Thehand model has an automatic grip. This feature saves the userJackfrom the independent manipulation of large numbers of joints and DOFs.The user speci(cid:12)es a grip type and an optional site on the ob ject to begrasped. When possible, the hand itself chooses a suitable approachdirection. Though frictional forces are not modeled, the positioningtask is greatly aided by the hand’s skill. Once gripped, the ob ject staysattached to the hand and moves along with it until explicitly freed bythe user.14CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUND1.3.3 A Human Model should be Sized According toPermissible Human DimensionsE(cid:11)ectively, there is no such thing as a \average" human. Statistically onemust always prescribe a target population when talking about percentiles ofsize, weight, or stature. A person may be 50percentile in stature, 75ththpercentile in weight, but 40percentile in lower leg length. Human dimen-thsional variability is enormous but not arbitrary. Within a given population,for example, 5percentile legs might never be found on anybody with 95ththpercentile arms, even though the population allows such sizes individually.Moreover, dimensions are not just limited to lengths, stature, and weight,but include joint limits, moments of inertia for each body segment, musclestrengths, fatigue rates, and so on. For proper behaviors, one must be ableto instantiate a properly sized (cid:12)gure with appropriately scaled attributes,preferably from a known population suitable for the required task analysis.1. Theanthropometric database is not proprietary. All data is readilyJackavailable and accessible. Consequently, it is easily customized to newpopulations or sets of individuals. Some databases are available, suchas NASA astronaut trainees, Army soldiers, and Society of AutomotiveEngineers \standard people."2. A database may consist of either population statistics or individuals.If populations, then percentile data points are expected in order to de-(cid:12)ne body dimensions. If individuals, then explicit information for eachperson in the collection is separately stored. For example, the NASAastronaut trainees constitute an explicit list of individuals, while theArmy soldier data is statistically derived.3. Enough information about a human (cid:12)gure must be stored to permitbody sizing, display, and motion. We use overall segment dimensions(length, width, thickness), joint limits, mass, moment of inertia, andstrength. Geometric properties are used during graphical manipulation;physical properties aid active behaviors (balance), dynamic simulation,and strength guided motion.4. With so many DOFs in a body and many useful physical attributes,there must be a convenient way of accessing, selecting, and modifyingany data.uses a spreadsheet-like interface to access anthropometryJackdata. This paradigm permits simple access and data interdependencies:for example, changing leg length should change stature; changing pop-ulation percentile should change mass distribution.5. Seeing the results of changing body dimensions is important to under-standing how di(cid:11)erent bodies (cid:12)t, reach, and see in the same workplace.Jackallows interactive body sizing while the body itself is under activeconstraints. Changing a (cid:12)gure seated in a cockpit from 95percentile toth1.3. WHAT IS CURRENTLY POSSIBLE?15th5percentile, for example, creates interesting changes in foot position,arm postures and view.1.3.4 A Human Model should have a Human-Like Ap-pearanceAs we argued earlier, a human model’s appearance has a lot to do with ourperception of acceptable behavior. The more accurate the model, the betterthe motions ought to be. Providing a selection of body models is a convenientway to handle a spectrum of interactive analysis and animation requirements.For quick assessments, a human model with simpli(cid:12)ed appearance might be(cid:12)ne. If the skin surface is not totally realistic, the designer can move the viewaround to check for su(cid:14)cient clearances, for example. When the completedanalysis is shown to the boss, however, an accurate skin model might be usedso that the robotic nature of the simpler model does not obscure the message.Looking better is often associated (in computer graphics) with being better.1. The \standard" or default body model instrikes a balance betweenJackdetail and interactive manipulation speed.It appears solid, has a 17segment (cid:13)exible torso, has a reasonable shoulder/clavicle mass, has afull hand, and has a generic face. A hat is added to avoid modelinghairstyles. The (cid:12)gure is modeled by surface polygons to take advantageof the available workstation display capabilities.2. There are times when more accurate skin surface models are needed. Forthat, computerized models of real people are used. These are derivedfrom a database of biostereometrically-scanned bodies.3. For extra realism, clothing is added to body segments by expanding andcoloring the existing segment geometry. Besides the inherent desirabilityof having a virtual (cid:12)gure in a work environment appear to be dressed,clothing will also a(cid:11)ect task performance if adjustments are made tojoint limits or if collision tests are performed.4. Facial features may be provided by graphical texture maps. By showinga speci(cid:12)c face a particular individual may be installed in the scene, the(cid:12)gure may be uniquely identi(cid:12)ed throughout an animation, or genericgender may be conveyed. Moreover, if the face is animated, an entirecommunication channel is enabled.1.3.5 A Human Model must Exist, Work, Act and ReactWithin a 3D Virtual EnvironmentWe do not live in Flatland [Abb53] and neither should virtual (cid:12)gures. Three-dimensional environment modeling is the hallmark of contemporary computer-aided design (CAD) systems. The workplace will frequently be constructedelectronically for design, analysis, and manufacturing reasons; we merely add16CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDhuman factors analysis to the list. Design importation facilitates up-frontanalyses before commitment to production hardware. Since geometric modelsare readily constructed, we must be sure that our body models and interactivesoftware are compatible with the vast ma jority of CAD modeling schemes sothe two can work together in the same virtual space.1. Sincemanipulates surface polygon geometry, simple features areJackprovided to interactively construct and edit the geometry of the virtualworkplace.is not intended as a substitute for a good CAD sys-Jacktem, but CAD features are provided for convenience. For example, ifa designer (cid:12)nds a problem with the imported workplace, the o(cid:11)endingregion can be immediately edited in. When the changes proveJackacceptable, the designer can use the data values fromto modifyJackthe \real" model in the external CAD system. While not optimal, thisavoids any tendency to migrate the working model intoand bypassJackthe more generalized CAD features provided by most CAD systems.2. Standardized geometric data transfer to and fromwould be highlyJackdesirable, but the state of standards in geometric modeling still leavessome important gaps. For example, few CAD systems adequately modelarticulated ob jects. Currently,imports models from several com-Jackmon CAD vendors through a less satisfactory scheme of explicit trans-lators from external geometry (cid:12)les. This situation will change to astandard as soon as possible.1.3.6 Use the Computer to Analyze Synthetic BehaviorsWhat would a real person do in a real environment? How can we get a virtualhuman to behave appropriately and report similar experiences? People areconstantly managing multiple simultaneous constraints or tasks, for example,staying balanced while reaching to lift a box, or walking while carrying a cupof co(cid:11)ee. The essential parallelism of human motion demands an approach tobehavior animation that does not just cope with parallelism but exploits it.We will be interested mostly in psychomotor and viewing behaviors; clearlyauditory and cognitive tasks are worthy of inclusion but are not dealt withhere.1.allows the user to specify, and the system maintains, multiple si-Jackmultaneous position and orientation goals. For instance, a typical pos-ture might involve reaching with two hands while looking at a targetand staying balanced. There are constraints on the positions of thehands and the orientation of the eyes dictated by the task, and bal-ance constraints required by gravity. Additionally, we might want thetorso to remain upright or, if seated, for the pelvis to tilt to a more re-laxed posture. There are too many possible human body con(cid:12)gurationsto manage every combination by specialized rules; our approach is to1.3. WHAT IS CURRENTLY POSSIBLE?17use a global solution technique,, to satisfy the giveninverse kinematicsconstraints sub ject to the inherent joint limits of the body.2. While in a posture, the strength requirements of the (cid:12)gure may bedisplayed on screen. This interactive strength data display shows alltorque loads along any selected chain of body segments.If a load isattached to an end-e(cid:11)ector, for example, it permits the easy visualizationof the distribution of that additional weight on the body.3. What the (cid:12)gure is looking at is often a critical question in human fac-tors analysis. In, the direction of eye gaze is controlled throughJackconstraints to some environmental location or ob ject site. If that sitemoves, the gaze will follow. Since eye movement will a(cid:11)ect head orienta-tion, the e(cid:11)ect of gaze direction can propagate (because of joint limits)to the neck and torso and hence in(cid:13)uence overall body posture.4. In, the user can see what the (cid:12)gure sees from an internal or exter-Jacknal perspective. Internally, a separate graphics window may be openedwhich shows the view from the selected eye. The image appears nat-urally shaded and it moves as the (cid:12)gure’s gaze is adjusted by directmanipulation or constraints. Concurrent changes to the environmentand visible parts of the (cid:12)gure’s own \self " are displayed in the viewwindow.If (cid:12)eld of view is critical, amay be usedretinal projectionwhere a polar pro jection displays workplace features based on their an-gle from the fovea. Although the image thereby appears distorted, theactual (cid:12)eld of view may be superimposed to assess the range of fovealor peripheral perception. For the external perspective, a selected (cid:12)eldof view is displayed as a translucent pair of view cones, one for eacheye. The cones move with the eyes. Ob jects in view are shadowed bythe translucent cones. Any overlapped region is clearly in the area ofbinocular vision.5. It is not yet feasible to do real-time collision detection between all themoving ob jects in a complex environment. By various simpli(cid:12)cations,however, su(cid:14)cient capabilities may be presented. On fast displays withreal-time viewing rotation (such as the Silicon Graphics workstations),the ability to rapidly change the view means that the user can quicklymove about to check clearances. Another method used inis toJackoptionally pro ject three orthogonal views of a (cid:12)gure and other selectedob jects onto back, side, and bottom planes. These three additionalviews give simultaneous contact and interference information and areused during interactive manipulations. Often, users will work withshaded images and detect collisions by noting when one ob ject visu-ally passes into another. The alternative to visual collision detectionis direct ob ject{ob ject interference computation. Usually limited to aselected body segment and a convex ob ject, this method is slower butguarantees to detect a collision even if it would be di(cid:14)cult to see.18CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUND6. Sometimes it is valuable to visualize the entire reachable space of anend-e(cid:11)ector of the (cid:12)gure. Empirical data is sometimes available in theliterature [NAS78, NAS87], but it is collected on speci(cid:12)c sub jects anddoes not readily extend to (cid:12)gures with di(cid:11)ering anthropometry or jointlimits. By constructing the reach space for a given (cid:12)gure in a givenposture as a geometric ob ject, the reach space may be viewed and ob jectsmay be readily classi(cid:12)ed as in or out of the reach space.1.3.7 An Interactive Software Tool must be Designed forUsabilityA system to build, move, and analyze virtual humans should be usable by meremortals with a modest training period. Isolating the potential user communityby requiring unusual artistic skills would be counter-productive to our widerpurposes of aiding design engineers. Existing interaction paradigms (such aspop-up menus or command line completions) should be followed when theyare the most e(cid:14)cacious for a particular task, but new techniques will beneeded to manage and control three-dimensional articulated structures withstandard graphical input tools. The interface should be simple yet powerful,comprehensive but easy to learn.1. Theuser interface is designed for fast response to multiple con-Jackstraint situations. Real-time end-e(cid:11)ector interactive dragging througharbitrary length joint chains means that the user can watch the (cid:12)gurerespond to reach or movement tasks. The paradigm of manipulating onejoint angle at a time is possible, but almost useless. The goal-directedbehaviors provide an enormous bene(cid:12)t to the user in allowing the spec-i(cid:12)cation ofis to be done while constraint satisfaction handles thewhathowthrough positioning interdependencies of the entire body structure.Dragging also permits quick experimentation with and manual optimiza-tion of postures.2. By taking advantage of the Silicon Graphics display hardware,Jackshows the user shaded or wireframe displays during interaction for nat-ural images and easy real-time visualization.3. For the highest quality images,provides its own multi-featuredJackray-tracing and radiosity programs.4. Besides direct user input,may be controlled through scripted com-Jackmands (in the) built in the course of interactiveJack command languagemanipulation. This saves time and trouble in setting up complex situa-tions, establishing a body posture, or trying a series of actions.5.also allows external control through operating system \sockets" toJackother programs, simulations, or real sensors, providing hooks into exter-nal data sources for virtual environments or networked remote systems1.4. MANIPULATION, ANIMATION, AND SIMULATION19sharing a common virtual dataspace.itself can share an environ-Jackment with other’s on the network. This could be used for novelJackcooperative workgroup applications.6. The standarduser interface consists of just a three button mouseJackand keyboard.It is simple to learn and no special hardware devicesare required unless a virtual environment setup is desired. The interac-tion paradigms ininclude menu-driven or typed commands, on-lineJackhelp, and command completion. It is easy to use after only a day or soof training.7. The direct manipulation interface into three dimensions implementedinis both highly e(cid:14)cient and natural to use. Depending onlyJackon the mouse and keyboard, it o(cid:11)ers a friendly, kinesthetic correspon-dence between manually comfortable hand motions, on-screen displays,and three-dimensional consequences. Three dimensional cursors, rota-tion wheels, and orthogonal pro jections of the principal view provideexcellent visual feedback to the user.8.provides multiple windows with independent camera views for com-Jackplex analyses, multiple points of view, and internal and external eyeviews.1.4 Manipulation, Animation, and SimulationThere are important distinctions between,, andmanipulationanimationsim-ulation. Geometric manipulation is the process of interactive scene composi-tion, or the interactive speci(cid:12)cation of positions and postures for geometric(cid:12)gures, usually on a trial and error basis. Manipulation usually involves move-ment of the (cid:12)gures, but the movement serves to assist in the control processand is generally not worth saving as a memorable motion sequence. Rather,the purpose of manipulation is to get the (cid:12)gures into a desired static posture,although the posture need not remain static afterwards. Manipulation is in-herently real-time: ob jects move as a direct response to the actions of theuser. In short, interactive manipulation is not necessarily choreography.Animation, on the other hand, is choreography. In computer animation,the goal is to describe motion, and the animator usually imagines the desiredmotion before beginning the animation process. Of course, experimentationmay lead to revisions, like an illustrator who erases lines in a drawing, butthe computer does not serve so much to answer questions as to obey orders.Animators measure the success of a computer animation system in terms ofhow well it serves as a medium for expressing ideas.Simulation is automated animation, and the concern is again with motion.The system generates the motion based on some kind of input from the userahead of time. The input usually consists of ob jectives and rules for makingdecisions, and it is generally less speci(cid:12)c than with animation. The user knows20CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUNDFigure 1.1: Is it the Motion or the Posture?less about what motion should result. The job of the simulator is to predictwhat would happen under certain circumstances and inform the user of theresults. Sometimes simulation can generate animation, as in the case of theanimation of physics and natural phenomena. Simulation of human (cid:12)guresgenerally implies some modeling of human capabilities to deliberately o(cid:11)-loadsome of the low-level positioning overhead from the animator.Animation and simulation have been studied extensively, but manipulationof articulated (cid:12)gures has not received the attention it deserves. Volumes ofresearch discuss animation techniques and simulation algorithms, but mostresearch directed at interactive manipulation deals either with the low-levelinput mechanisms of describing 3D translations and rotations, or with thenumerical issues of real-time dynamics. For example, consider the task ofbending a (cid:12)gure over to touch its toes (Fig. 1.1.Is the bending motionimportant, or is it just the (cid:12)nal posture that is critical? In animation, it’sthe motion: the motion must look realistic. In simulation, the motion mustberealistic. In manipulation, the (cid:12)ner points of the posture are critical. Isthe (cid:12)gure balanced? Are the knees bent? Where is the head pointed? Howare the feet oriented? The motion through which the manipulation systempositions the (cid:12)gure is not important in itself. It serves only to assist the userin arriving at the posture.1.5 What Did We Leave Out?It is only fair that in this exposition we are clear about what our existingsoftware does not do. There are choices to be made in any implementation, but1.5. WHAT DID WE LEAVE OUT?21the vastness of the human performance problem demands scope boundariesas well. There are fascinating problems remaining that we have not touched.Some of these problems are being examined now, but the early results aretoo premature to report. The activity in this (cid:12)eld is amazing, and therewill surely be advances in modeling, animation, and performance simulationreported each year.A glance at the[BL88], for example, willEngineering Data Compendiumquickly show how much information has been collected on human factors andsimultaneously how little is available interactively on a computer. But evenwithout much thought, we can place some bounds on this study.(cid:15)We have ignored auditory information processing, environmental fac-tors (such as temperature and humidity), vibration sensitivity, and soon. While critical for harsh environments, we seek useful approxima-tions to (cid:12)rst order (geometric) problems to see if a (cid:12)gure can do thetask in the absence of external signals, distress, or threats. The proba-ble degradation of task performance may be established from intelligent(manual) search through existing publications. While it would be at-tractive to include such data, we have not begun to acquire or use ityet.geometry.(cid:15)We have bypassed super-accurate skin models because the en(cid:13)eshmentof a (cid:12)gure is so dependent on individual physiology, muscle tone, mus-cle/fat ratio, and gender. For the kinds of analyses done with wholebody models, errors of a centimeter or so in skin surfaces are subordi-nate to anthropometric errors in locating true joint centers and joint(cid:15)We have avoided injury assessment, as that is another whole (cid:12)eld devel-oped from anatomy, crash studies, or national exertion standards. Ofcourse,could be used in conjunction with such systems, but weJackhave not tried to connect them yet.(cid:15)Because we have taken a generalized view of human factors and re-stricted analyses to reach, (cid:12)t, view, and strength, we have necessarilyavoided any workplace-speci(cid:12)c performance data. For example, actiontimings will be most accurate when measured with real sub jects in areal environment. We have concentrated on measurements in environ-ments prior to physical construction to save cost and possible personneldangers.(cid:15)We have only touched on perceptual, reactive and cognitive issues. Manyworkers are engaged in control-theoretic activities: sensing the environ-ment and reacting to maintain some desired state. We see this capabil-ity eventually driving the (cid:12)gure through the simulation interface, but atthe present we have not modeled such situations. Instead we are investi-gating natural language instructions which generate an animation of thesimulated agent’s \arti(cid:12)cial intelligence" understanding of the situation.22CHAPTER 1.INTRODUCTION AND HISTORICAL BACKGROUND(cid:15)Finally, we are leaving issues of learning for later study. There is much tobe said for an agent who not only follows instructions but who also learnshow to do similar (but not identical) actions in comparable situationsin the future. Learning might (cid:12)rst occur at the psychomotor level,for example, to (cid:12)gure out the best way to lift a heavy weight. Later,learning can extend to the task level: to repeat a variant of a previouslysuccessful plan when the overall goals are encountered again.These issues are exciting, but we need to start with the basics and describewhat exists today.Chapter 2Body ModelingIn order to manipulate and animate a human (cid:12)gure with computer graphics,a suitable (cid:12)gure must be. This entails constructing a satisfactorymodeledsurface skin for the overall human body shape, de(cid:12)ning a skeletal structurewhich admits proper joint motions, adding clothing to improve the verisimil-itude of analyses (as well as providing an appropriate measure of modesty),sizing body dimensions according to some target individual or population,and providing visualization tools to show physically-relevant body attributessuch as torque loads and strength.2.1 Geometric Body ModelingIn computer graphics, the designer gets a wide choice of representations forthe surfaces or volumes of ob jects. We will brie(cid:13)y review current geometricmodeling schemes with an emphasis on their relevance to human (cid:12)gures.We classify geometric models into two broad categories: boundary schemesand volumetric schemes.In a boundary representation the surface of theob ject is approximated by or partitioned into (non-overlapping) 0-, 1-, or 2-dimensional primitives. We will examine in turn those representations relevantto human modeling: points and lines, polygons, and curved surface patches.In a volumetric representation the 3D volume of the ob ject is decomposedinto (possibly overlapping) primitive volumes. Under volumetric schemes wediscuss voxels, constructive solid geometry, ellipsoids, cylinders, spheres, andpotential functions.2.1.1 Surface and Boundary ModelsThe simplest surface model is just a collection of 3D points or lines. Surfacesrepresented by points require a fairly dense distribution of points for accuratemodeling. Clouds of points with depth shading were used until the early1980’s for human models on vector graphics displays. They took advantage of2324CHAPTER 2. BODY MODELINGthe display’s speed and hierarchical transformations to produce the perceptualdepth e(cid:11)ect triggered by moving points [Joh76] (for example, [GM86]).A related technique to retain display speed while o(cid:11)ering more shape in-formation is to use parallel rings or strips of points. This technique is usedin[Lif91, Cal91]. Artistically positioned \sketch lines" wereLif eF ormsTM1used in one of the earliest human (cid:12)gure models [Fet82] and subsequently in aMick Jagger music video, \Hard Woman" from Digital Productions.PolygonsPolygonal (polyhedral) models are one of the most commonly encounteredrepresentations in computer graphics. The models are de(cid:12)ned as networks ofpolygons forming 3D polyhedra. Each polygon (primitive) consists of someconnected vertex, edge, and face structure. The polygons are sized, shaped,and positioned so that they completely tile the required surface at some res-olution. Polygon models are relatively simple to de(cid:12)ne, manipulate, and dis-play. They are the most common models processed by workstation hardwareand commercial graphics software. In general, polygons are best at model-ing ob jects meant to have (cid:13)at surfaces, though with a large enough numberof polygons quite intricate and complex ob jects can be represented. \Largeenough" may mean hundreds of thousands of polygons!All viablehuman (cid:12)gure models are done with polygons, pri-interactivemarily because the polygon is the primitive easiest to manage for a modernworkstation such as the Silicon Graphics machines. With real-time smoothshading, polygon models of moderate complexity (several hundred polygons)can look acceptably human-like; accurate skin models require thousands. Arealistic face alone may require two or three thousand polygons if it is to beanimated and if the polygons are the only source of detail.Polygon models are too numerous to cite extensively. Ones with an inter-esting level of detail have been used in Mannequin software from Biomechan-ics Corporation of America [Pot91], various movies such as \Tony de Peltrie"from the University of Montreal [Emm85], and detailed synthetic actor modelsof Marilyn Monroe and Humphrey Bogart from Daniel Thalmann and NadiaMagnenat-Thalmann [MTT90, MTT91b]. They even model details of skin de-formation by applying physical forces to polygon meshs [GMTT89, MTT91b].The polygon models used inare polygonal with two di(cid:11)erent levels ofJackdetail. The normal models have a few hundred polygons. More accurate mod-els obtained from actual scans of real bodies have several thousand polygons.(See Section 2.1.3.)Curved SurfacesSince polygons are good at representing (cid:13)at surfaces, considerable e(cid:11)ort hasbeen expended determining mathematical formulations for true curved sur-faces. Most curved surface ob ject models are formed by one or more para-1LifeForms is a registered trademark of Kinetic E(cid:11)ects, Inc.2.1. GEOMETRIC BODY MODELING25metric functions of two variables (bivariate functions). Each curved surfaceis called a patch; patches may be joined along their boundary edges intomore complex surfaces. Usually patches are de(cid:12)ned by low order polynomials(typically cubics) giving the patch easily computed mathematical propertiessuch as well-de(cid:12)ned surface normals and tangents, and computable continuityconditions between edge-adjacent patches. The shape of a patch is derivedfrom control points or tangent vectors; there are both approximating andinterpolating types. The former take the approximate shape of the controlvertices; the latter must pass through them. There are numerous formulationsof curved surfaces, including: Bezier, Hermite, bi-cubic, B-spline, Beta-spline,and rational polynomial [Far88, BBB87].Various human (cid:12)gure models have been constructed from curved patches,but display algorithm constraints make these (cid:12)gures awkward for real-timemanipulation. They are excellent for animation. provided that su(cid:14)cient careis taken to model joint connections. This is a good example of where increasedrealism in the body segments demands additional e(cid:11)ort in smoothing andbending joint areas properly. Curved surface models were used in a gymnasticpiece [NHK86] and the Academy Award-winning \Tin Toy" [GP88].2.1.2 Volume and CSG ModelsThe volume and CSG models divide the world into three-dimensional chunks.The models may be composed of non-intersecting elements within a spatialpartition, such as voxels or oct-trees, or created from (possibly overlapping)combinations of inherently 3D primitive volumes.Voxel ModelsThe (cid:12)rst volumetric model we examine is the voxel model. Here space iscompletely (cid:12)lled by a tessellation of cubes or parallelopipeds called voxels(volume elements). Usually there is a density or other numerical value asso-ciated with each voxel. Storing a high resolution tessellation is expensive inspace but simple in data structure (just a large 3D array of values). Usuallysome storage optimization schemes are required for detailed work (1K x 1Kx 1K spaces). Special techniques are needed to compute surface normals andshading to suppress the boxiness of the raw voxel primitive. Voxel data iscommonly obtained in the medical domain; it is highly regarded for diagnos-tic purposes as the 3D model does not speculate on additional data (say bysurface (cid:12)tting) nor suppress any of the original data however convoluted.Voxel models are the basis for much of the scienti(cid:12)c visualization workin biomedical imaging [FLP89]. The possible detail for human models isonly limited by the resolution of the sensor. Accurate bone joint shapes maybe visualized, as well as the details of internal and external physiologicalfeatures. These methods have not yet found direct application in the humanfactors domain, since biomechanical rather than anatomical issues are usuallyaddressed. Real-time display of voxel images is also di(cid:14)cult, requiring either26CHAPTER 2. BODY MODELINGlow resolution image sets or special hardware [GRB85].+Constructive Solid GeometryOne of the most e(cid:14)cient and powerful modeling techniques is constructivesolid geometry (CSG). Unlike the voxel models, there is no requirement toregularly tessellate the entire space. Moreover, the primitive ob jects are notlimited to (uniform) cubes; rather there are any number of simple primitivessuch as cube, sphere, cylinder, cone, half-space, etc. Each primitive is trans-formed or deformed and positioned in space. Combinations of primitives or ofpreviously combined ob jects are created by the Boolean operations. An ob-ject therefore exists as a tree structure which is \evaluated" during renderingor measurement.CSG has been used to great advantage in modeling machined parts, buthas not been seriously used for human body modeling. Besides the mechanicallook created, real-time display is not possible unless the CSG primitives arepolygonized into surfaces. When the set of primitives is restricted in one wayor other, however, some useful or interesting human models have been built.Single Primitive SystemsThe generality of the constructive solid geometry method { with its multiplic-ity of primitive ob jects and expensive and slow ray-tracing display method {is frequently reduced to gain e(cid:14)ciency in model construction, avoid Booleancombinations other than union, and increase display speed. The idea is to re-strict primitives to one type then design manipulation and display algorithmsto take advantage of the uniformity of the representation. Voxels might beconsidered such a special case, where the primitives are all coordinate axisaligned and integrally positioned cubes. Other schemes are possible, for ex-ample, using ellipsoids, cylinders, superquadrics, or spheres.Ellipsoids have been used to model cartoon-like (cid:12)gures [HE78, HE82].They are good for elongated, symmetric, rounded ob jects. Unfortunately, theshaded display algorithm is nearly the same as the general ray-tracing process.Cylinders have also been used to model elongated, symmetric ob jects. El-liptic cylinders were used in an early human modeling system [Wil82]. Theseprimitives su(cid:11)er from joint connection problems and rather poor representa-tions of actual body segment cross-sections.Superquadrics are a mathematical generalization of spheres which includean interesting class of shapes within a single framework: spheres, ellipsoids,and ob jects which arbitrarily closely look like prisms, cylinders, and stars.Simple parameters control the shape so that deformations through membersof the class are simple and natural. Superquadrics are primarily used to modelman-made ob jects, but when overlapped can give the appearance of faces and(cid:12)gures [Pen86].Spheres as a single primitive form an intriguing class. Spheres have asimplicity of geometry that rivals that of simple points: just add a radius.2.1. GEOMETRIC BODY MODELING27There are two methods of rendering spheres. Normally they are drawn asregular 3D ob jects. A human modeled this way tends to look like a largebumpy molecule. Alternatively, spheres may be treated like \scales" on themodeled ob ject; in this case a sphere is rendered as a (cid:13)at shaded disk. Withsu(cid:14)cient density of overlapping spheres, the result is a smoothly shaded solidwhich models curved volumes rather well. A naturalistic human (cid:12)gure wasdone this way in our earlier TEMPUS system [BB78, BKK85, SEL84]. We+stopped using this method as we could not adequately control the sphere/diskoverlaps during animation and newer workstation display technology favoredpolygons.Potential FunctionsAn interesting generalization of spheres which solves some ma jor modelingproblems is to consider the volume as a potential function with a center and a(cid:12)eld function that decreases monotonically (by an exponential or polynomialfunction) from the center outward. There is no \radius" or size of the potentialfunction; rather, the size or surface is determined by setting a threshold valuefor the (cid:12)eld. What makes this more interesting is that potential functions actlike energy sources: adjacent potential functions have overlapping (cid:12)elds andthe resultant value at a point in space is in fact the sum of the (cid:12)elds active atthat point. Thus adjacent (cid:12)elds blend smoothly, unlike the \creases" that areobtained with (cid:12)xed radius spheres [Bli82]. Recently, directional dependenceand selective (cid:12)eld summation across models have been added to create \soft"models that blend with themselves but not with other modeled ob jects in theenvironment [WMW86, NHK85]. Potential functions were originally used+to model molecules, since atoms exhibit exactly this form of (cid:12)eld behavior,but the models have an amazing naturalistic \look" and have been used togreat e(cid:11)ect in modeling organic forms including human and animal (cid:12)gures[NHK85, BS91]. The principal disadvantages to potential functions lie inproperly generating the numerous overlapping functions and very slow displaytimes. They remain an interesting possibility for highly realistic models in the2.1.3 The Principal Body Models UsedThe default polyhedral human (cid:12)gure inis composed of 69 segments,Jack68 joints, 136 DOFs, and 1183 polygons (including cap and glasses). Theappearance is a compromise between realism and display speed. No one islikely to mistake the (cid:12)gure for a real person; on the other hand, the movementsand speed of control are good enough to convey a suitably responsive attitude.The stylized face, hat, and glasses lend a bit of character and actually assistin the perception of the forward-facing direction.For more accurate human bodies, we have adapted a database of actualbody scans of 89 sub jects (31 males and 58 females) supplied by Kathleen2Pei-Hwa Ho.+future.228CHAPTER 2. BODY MODELINGRobinette of Wright-Patterson Air Force Base and used with her permission.The original data came in contours, that is, slices of the body in the trans-verse plane [GQO89]. Each body segment was supplied as a separate set of+contours. A polygon tiling program was used to transform the contours ofeach body segment into a surface representation.In order to represent the human body as an articulated (cid:12)gure we needed to(cid:12)rst compute the proper joint centers to connect the segments together. Jointcenter locations were computed through the coordinates of anthropometriclandmarks provided with the contour data.Real humans are not symmetrical around the sagittal plane: our left halfis not identical to our right half. This was the case with the contour data.For consistency, rather than accuracy, we used the right half of the body datato construct a left half and then put them together. We also sliced the uppertorso to take advantage of the seventeen segment spine model (Section 2.3).The resulting human body model has thirty-nine segments and about 18,700polygons compared to the 290 slices in the original data.2.2 Representing Articulated FiguresUnderneath the skin of a human body model is a representation of the skele-ton. This skeletal representation serves to de(cid:12)ne the moving parts of the(cid:12)gure. Although it is possible to model each of the bones in the human bodyand encode in the model how they move relative to each other, for most typesof geometric analyses it is su(cid:14)cient to model the body segments in termsof their lengths and dimensions, and the joints in terms of simple rotations.There are some more complex joint groups such as the shoulder and spinewhere inherent dependencies across several joints require more careful andsophisticated modeling.The increasing interest in recent years in ob ject-oriented systems is largelydue to the realization that the design of a system must begin with a deepunderstanding of the ob jects it manipulates. This seems particularly true in ageometric modeling system, where the word \ob ject" takes on many of its lessabstract connotations. It has long been an adage in the user interface softwarecommunity that a system with a poorly designed basic structure cannot berepaired by improving the interface, and likewise that a well designed systemlends itself easily to an elegant interface.This section describes, which represents articulatedcom-Peabody(cid:12)guresposed ofconnected by. Thedata structure has asegmentsjointsPeabodycompanion language and an interactive interface infor specifying andJackcreating articulated (cid:12)gures. The data structure itself maintains geometric in-formation about segment dimensions and joint angles, but it also provides ahighly e(cid:14)cient mechanism for computing, storing, and accessing various kindsof geometric information. One of the principal tasks requested ofPeabodyis to map segment dimensions and joint angles into global coordinates for ende(cid:11)ectors.2.2. REPRESENTING ARTICULATED FIGURES29Peabodywas designed with several criteria in mind:(cid:15)It should be general purpose. It should be able to represent many typesof (cid:12)gures of tree-structured topology. It should not be hard coded torepresent a speci(cid:12)c type of (cid:12)gure, such as a human (cid:12)gure or a particularrobot manipulator.(cid:15)It should have a well developed notion of articulation. Rather than con-centrating on representations for primitive geometric shapes,Peabodyaddresses how such shapes can be connected together and how theybehave relative to each other.(cid:15)It should represent tree-structured ob jects through a hierarchy. Theinverse kinematics positioning algorithm can calculate and maintain theinformation necessary to simulate closed loops.(cid:15)It should be easy to use. The external user view of the (cid:12)gures should belogical, clear, and easy to understand. Understanding the (cid:12)gures shouldnot require any knowledge of the internal implementation, and it shouldnot require any advanced knowledge of robotics or mechanics.2.2.1 BackgroundKinematic Notations in RoboticsThe most common kinematic representation in robotics is the notation ofDenevit and Hartenberg [Pau81, DH55]. This representation derives a set ofparameters for describing a linkage based on measurements between the axesof a robot manipulator. The notation de(cid:12)nes four parameters that measurethe o(cid:11)set between subsequent coordinate frames embedded in the links, orsegments: 1) the angle of rotation for a rotational joint or distance of trans-lation for a prismatic joint; 2) the length of the link, or the distance betweenthe axes at each end of a link along the common normal; 3) the lateral o(cid:11)setof the link, or the distance along the length of the axis between subsequentcommon normals; and 4) the twist of the link, or the angle between neigh-boring axes. The notation prescribes a formal procedure for assigning thecoordinate systems to the links in a unique way.The ob jective behind these kinematic notations in robotics is to developa standard representation that all researchers can use in the analysis anddescription of manipulators. There are several types of manipulators that areextremely common in the robotics research community. The adoption of astandard representation would greatly simplify the process of analyzing andimplementing robotics algorithms since so many algorithms are described inthe literature using these manipulators.Animation SystemsComputer graphics and animation literature seldom addresses syntactic, oreven semantic, issues in representations for mechanisms, except as background30CHAPTER 2. BODY MODELINGfor some other discussion of an animation technique or system.Most interactive animation systems such as GRAMPS [OO81], TWIXT[Gom84], and BBOP [Stu84, Ste83], as well as commercial animation packagessuch as Alias [Ali90] and Wavefront [Wav89] only provide a mechanism of at-taching one ob ject to another. In this way, the user can construct hierarchies.When the user manipulates one ob ject, its child ob jects follow, but there isno real notion of articulation. The attachments simply state that the originof the child ob ject is relative to the origin of the parent.Many animation systems are non-interactive and are based on scripts thatprovide a hierarchy only through a programming language interface. Exam-ples of such systems are ANIMA-II [Hac77], ASAS [Rey82] and MIRA-3D[MTT85]. In this kind of system, the hierarchy is hard-coded into the script,possibly through an interaction loop. A hierarchy designed in this way isvery limited, except in the hands of a talented programmer/animator whocan write into the animation a notion of behavior.Physically-Based Modeling SystemsPhysically based modeling systems such as that of Witkin, Fleisher, and Barr[WFB87] and Barzel and Barr [BB88] view the world as ob jects and con-straints. Constraints connect ob jects together through desired geometric re-lationships or keep them in place. Otherwise, they (cid:13)oat in space under theappropriate laws of physics. There is no notion of articulation other thanconstraints. This forces the burden of maintaining ob ject positions entirely tothe algorithms that do the positioning. For simple ob jects, this is conceptu-ally pleasing, although for complex ob jects it is computationally di(cid:14)cult. Ifa system represents joints like the elbow as a constraint, the constraint musthave a very high weighting factor in order to ensure that it never separates,requiring very small time steps in the simulation. This may also complicatethe user’s view of ob jects such as robots or human (cid:12)gures, which are inher-ently articulated. We believe it is important to di(cid:11)erentiate the relationshipbetween body segments at the elbow and the relationship between a hand anda steering wheel.2.2.2 The Terminology of PeabodyPeabodyenvironmentuses the termto refer to the entire world of geometricob jects. The environment consists of individual, each of which is a(cid:12)gurescollection of. The segments are the basic building blocks of thesegmentsenvironment. Each segment has a geometry. It represents a single physicalob ject or part, which has shape and mass but no movable components. Thegeometry of each segment is represented by a, which is generally apsurfpolyhedron or a polygonal mesh but can be of a more general nature.The termapplies not only to articulated, jointed (cid:12)gures such as a(cid:12)gurehuman body: any single \ob ject" is a (cid:12)gure. It need not have moving parts.A (cid:12)gure may have only a single segment, such as a co(cid:11)ee cup, or it may be2.2. REPRESENTING ARTICULATED FIGURES31composed of several segments connected bysuch as a robot. Sometimesjoints,the term \ob ject" denotes any part of theenvironment.PeabodyJoints connect segments through attachment frames called. A site issitesa local coordinate frame relative to the coordinate frame of its segment. Eachsegment can have several sites. Joints connect sites on di(cid:11)erent segmentswithin the same (cid:12)gure. Sites need not lie on the surface of a segment. A siteis a coordinate frame that has an orientation as well as a position. Each sitehas athat is the homogeneous transform that describes its placementlocationrelative to the base coordinate frame of its segment.Segments do not have speci(cid:12)c dimensions, such as the length, o(cid:11)set, andtwist of Denevit and Hartenberg notation, because the origin can lie anywhereon the segment. The location of the axes of the joints that connect the seg-ment are phrased in terms of this origin, rather than the other way around.The measurement of quantities such as length is complicated, because seg-ments may have several joints connected to them, and none of these joints isdesignated in the de(cid:12)nition as the \parent."Joints may have several DOFs, which are rotational and translational axes.Each axis and its corresponding angle form a single rotation or translation,and the product of the transform at each DOF de(cid:12)nes the transform acrossthe joint, de(cid:12)ning the placement of the sites, and thus the segments, that thejoint connects.The directionality of a joint is important because it de(cid:12)nes the order inwhich the DOF transforms are concatenated. Because these transforms arenot commutative, it is essential that the order is well-de(cid:12)ned. This is anespecially important feature of, since it is sometimes convenient toPeabodyde(cid:12)ne the direction of the joint in a way di(cid:11)erent from the way the joint occursin the (cid:12)gure hierarchy. An example of this is the human knee. Although itmay be useful to structure the hierarchy of a human body with the root atthe foot, it is also appealing to have the joints at both knees de(cid:12)ned in thesame manner.2.2.3 The Peabody HierarchyPeabodyavoids imposing a prede(cid:12)ned hierarchy on the (cid:12)gures by encour-aging the user to think of (cid:12)gures as collections of segments and joints, nonewith special importance. However, there must exist an underlying hierarchybecauseis not equipped to handle closed-loop mechanisms. (ClosedPeabodyloop structures are managed through the constraint satisfaction mechanism.)The structure of thetree is de(cid:12)ned by designating one site on thePeabody(cid:12)gure as the. The root site roughly corresponds to the origin of the (cid:12)g-rooture, and it provides a handle by which to specify the location of the (cid:12)gure.Viewing the (cid:12)gure as a tree, the root of the (cid:12)gure is the root of the tree.The root site of a (cid:12)gure may change from time to time, depending upon thedesired behavior of the (cid:12)gure.This means there are two representations for the hierarchy, one internaland one external. There are many advantages to having a dual representation32CHAPTER 2. BODY MODELINGof the hierarchy. First, it allows the hierarchy to be inverted on the (cid:13)y. Mostmodels have a natural order to their hierarchy, emanating from a logical origin,but this hierarchy and origin may or may not correspond to how a model isplaced in the environment and used.The choice of the (cid:12)gure root is particularly important to the inverse kine-matics algorithm, since the algorithm operates on chains of joints within the(cid:12)gure. At least one point on the (cid:12)gure must remain (cid:12)xed in space. Becausethe internal representation of the hierarchy is separate, the user maintains aconsistent view of the transform across a joint, regardless of how the (cid:12)gure isrooted.The example below illustrates thehierarchy. Each segment hasPeabodyits base coordinate frame in the middle and an arc leading to each of its sites.The transform along this arc is the site’s location. Each site may have severaljoints branching out from it, connecting it downwards in the tree to sites onother segments.figure table {segment leg {psurf = "leg.pss";attribute = plum;site base->location = trans(0.00cm,0.00cm,0.00cm);site top->location = trans(5.00cm,75.00cm,5.00cm);segment leg0 {psurf = "leg.pss";attribute = springgreen;site base->location = trans(0.00cm,0.00cm,0.00cm);site top->location = trans(5.00cm,75.00cm,5.00cm);segment leg1 {psurf = "leg.pss";attribute = darkslategray;site base->location = trans(0.00cm,0.00cm,0.00cm);site top->location = trans(5.00cm,75.00cm,5.00cm);segment leg2 {psurf = "leg.pss";attribute = darkfirebrick;site base->location = trans(0.00cm,0.00cm,0.00cm);site top->location = trans(5.00cm,75.00cm,5.00cm);}}}}segment top {psurf = "cube.pss" * scale(1.00,0.10,2.00);site base->location = trans(0.00cm,0.00cm,0.00cm);site leg0->location = trans(95.00cm,0.00cm,5.00cm);site leg2->location = trans(5.00cm,0.00cm,5.00cm);2.2. REPRESENTING ARTICULATED FIGURES33site leg3->location = trans(95.00cm,0.00cm,195.00cm);site leg4->location = trans(5.00cm,0.00cm,195.00cm);}}}}}joint leg1 {connect top.leg3 to leg1.top;type = R(z);joint leg2 {connect top.leg2 to leg0.top;type = R(z);joint leg3 {connect top.leg4 to leg2.top;type = R(z);joint leg4 {connect top.leg0 to leg.top;type = R(z);root = top.base;location = trans(0.00cm,75.00cm,0.00cm);}32.2.4 Computing Global Coordinate TransformsThe root site for the (cid:12)gure is the one at the top of the tree, and its globallocation is taken as given, that is, not dependent on any other element ofthe environment. The root, the site locations, and the joint displacementsuniquely determine the global location of every site and segment in the treein terms of a product of transforms from the root downward.The computation of the coordinate transforms for each segment and sitein the downward traversal of the tree requires inverting the site locationsthat connect the segment to other segments lower in the tree. It may alsorequire inverting joint displacements if the joint is oriented upwards in the tree.Computationally, this is not expensive because the inverse of a homogeneoustransform is easy to compute, through a transpose and a dot product.2.2.5 Dependent JointsThe human (cid:12)gure can be abstracted as an ob ject which is to be instantiated(into a certain pose) by any speci(cid:12)cation of all the joint angles. While anypose can be represented by a set of joint angles, it is not always possible tosupply a full and reasonable set of angles. Often, for example, there is a3Jianmin Zhao.34CHAPTER 2. BODY MODELINGnatural grouping of joints such as the torso or shoulder mass that typicallywork together. Arbitrary (admissible) joint angles for the joints in the groupmay not represent a legitimate posture: they are functionally dependent oneach other.program.Conceptually, these dependencies compromise the notion of the joint andjoint angle and blur the boundary of the ob ject de(cid:12)nition. It seems that notall joints are created equal. Rather than have a system which tries to copewith every joint the same way, we take a more practical approach. We use ajoint groupPeabodyconcept into accommodate joint dependency so thatthe relationship is coded into the ob ject de(cid:12)nition rather than the applicationA joint group is a set of joints which are controlled as one entity. Internaljoint angles are not visible outside of the group: they are driven by the groupdriver. The driver is nothing but a mapping from a number of parameters(counterparts of joint angles of the independent joint) to joint angles of itsconstituent joints. Those independent parameters will be called.group anglesSimilar to the joint angles of the independent joint, the group angles of thejoint group are sub ject to linear constraints of the formn=1iXa(cid:18)b(cid:20)iii(21):where’s are group angles andis the number of’s, or number of DOFs of(cid:18)n(cid:18)the joint group. There may be many such constraints for each group.There can be many applications of the joint group. Forearm pronation andsupination change the segment geometry, so one way to manage that withinthe geometry constraints of psurfs is to divide the forearm into a number ofnearly cylindrical sub-segments. As the wrist moves, its rotation is trans-mitted to the forearm segments such that distal sub-segments rotate morethan proximal ones. The segment adjacent to the elbow does not pronate orsupinate at all. Fingers could be managed in a similar fashion by distributingthe desired orientation of the (cid:12)ngertip over the three joints in the (cid:12)nger chain.The most interesting examples, though, involve the torso and the shoulder.We address these cases in the next sections.2.3 A Flexible Torso Model4Human (cid:12)gure models have been studied in computer graphics almost sincethe introduction of the medium. Through the last dozen years or so, thestructure, (cid:13)exibility, and (cid:12)delity of human models has increased dramatically:from the wire-frame stick (cid:12)gure, through simple polyhedral models, to curvedsurfaces, and even (cid:12)nite element models. Computer graphics modelers havetried to maximize detail and realism while maintaining a reasonable overalldisplay cost. The same issue pertains to control: improving motion realism4Gary Monheit.2.3. A FLEXIBLE TORSO MODEL35requires a great number of DOFs in the body linkage, and such redundancystrains e(cid:11)ective and intuitively useful control methods. We can either sim-plify control by simplifying the model, thereby risking unrealistic movements;or complicate control with a complex model and hope the resulting motionsappear more natural. The recent history of computer animation of human(cid:12)gures is focused on the quest to move the technology from the former situ-ation towards the latter while simultaneously forcing the control complexityinto algorithms rather than skilled manual manipulation.This point of view motivates our e(cid:11)orts in human (cid:12)gure modeling andanimation, as well as those of several other groups. Though notable algorithmsfor greater animation power have addressed kinematics, dynamics, inversekinematics, available torque, global optimization, locomotion, deformation,and gestural and directional control, the human models themselves tendedto be rather simpli(cid:12)ed versions of real human (cid:13)exibility. In the early 1980’swe warned that increased realism in the models would demand ever moreaccurate and complicated motion control; now that the control regimes areimproving, we must return to the human models and ask if we must re-evaluatetheir structure to take advantage of algorithmic improvements. When weconsidered this question, we determined that a more accurate model of thehuman spine and torso would be essential to further realism in human motion.Although many models have appeared to have a (cid:13)exible torso, they havebeen computer constructions of the surface shape manipulated by skilled an-imators [Emm85]. We needed a torso that was suitable for animation, butalso satis(cid:12)ed our requirements for anthropometric scalability. Thus a singlemodel of (cid:12)xed proportions is unacceptable as human body types manifestconsiderable di(cid:11)erences. (A similar type of (cid:13)exible (cid:12)gure is found in snakes[Mil88, Mil91], but the anthropometry issues do not arise. Moreover, thissnake animation is dynamics-based; humans do not need to locomote by wig-gling their torsos and so a kinematics model was deemed adequate.) Zeltzerand Stredney’s \George" skeleton model has a detailed vertebral column, butit is not articulated nor is it bent during kinematic animation [Zel82]. Lim-ited neck vertebral motion in the saggital plane was simulated by Willmert[Wil82]. Various body models attempt a spine with a 3D curve but shape andcontrol it in afashion.ad hocIf the spine were realistically modeled, then the torso, a vessel connectedand totally dependent on the spine, could then be viewed and manipulatedinteractively. So we undertook the development of a far more satisfactory andhighly (cid:13)exible vertebral model of the spine and its associated torso shape.The conceptual model of the spinal column is derived from medical dataand heuristics related to human kinesiology. The spine is a collection of verte-brae connected by ligaments, small muscles, vertebral joints (called processes),and intervertebral discs [BBA88]. Nature has designed the spine for supportof the body’s weight, stability of the torso, (cid:13)exibility of motion, and protectionof the spinal cord [AM71, Hol82].The spine moves as a column of vertebrae connected by dependent joints,meaning that it is impossible to isolate movement of one vertebral joint from36CHAPTER 2. BODY MODELINGthe surrounding vertebrae [Lou83]. Muscle groups of the head, neck, abdomenand back initiate the movement of the spine, and the interconnecting ligamentsallow the movement of neighboring vertebrae [BBA88, Wel71].2.3.1 Motion of the SpineAnatomy of the Vertebrae and DiscThe spinal column consists of 33 vertebrae organized into 5 regions [BBA88]:cervical, thoracic, lumbar, sacral, and coccyx.The vertebrae are labeled by medical convention in vertical descendingorder: C1{C7, T1{T12, L1{L5, and S1{S5. Which regions should be consid-ered part of the torso? The cervical spine lies within the neck. The sacrumand coccyx contain vertebrae that are (cid:12)xed through fusion [AM71]. Since themobile part of the torso includes the 12 thoracic and 5 lumbar vertebrae, alltogether 17 vertebrae and 18 joints of movement are included in the torsomodel.Each vertebra is uniquely sized and shaped, but all vertebrae contain acolumnar body and an arch. The body is relatively large and cylindrical, sup-porting most of the weight of the entire spine. The vertebral bodies increasegradually in size from the cervical to the lumbar region [AM71].The arch supports seven processes: four articular, two transverse, and onespinous [AM71]. The processes are bony protrusions on the vertebra that aidand limit the vertebral motion. The transverse and spinous processes serveas levers for both muscles and ligaments [BBA88]. The articular processesprovide a joint facet for the joint between successive vertebral arches. Theseprocesses, due to their geometry, cause the vertebrae to rotate with 3 DOFs.Ligaments and small muscles span successive vertebral processes. They givethe spinal column its stability. Because of this strong interconnectivity, spinalmovement is modeled as interdependent movements of neighboring joints.Vertebrae are each separated by intervertebral discs. The disc has 3 parts[Lou83]:(cid:15)nucleus pulposus- the sphere in the center, consisting of 85% water(cid:15)annulus (cid:12)brosus- the (cid:12)bers running as concentric cylinders aroundthe nucleusbody.(cid:15)cartilaginous plates- a thin wall separating the disc from the vertebralThe disc changes shape as the neighboring vertebrae bend. But, since thenucleus is 85% water, there is very little compression. The disc can bulgeout spherically, as force is applied to the columnar body above or below.Therefore, overall the disc does not function as a spring, but as a deformablecylindrical separation between vertebrae, supporting the theory that the ver-tebrae do not slide, but rotate around an axis [Lou83].2.3. A FLEXIBLE TORSO MODEL37Range of Movement of Each VertebraVertebral movement is limited by the relative size of the disks, the attachedligaments, and the shape and slant of the processes and facet joints. Statis-tics for joint limits between each successive vertebra have been recorded andcompiled [Lou83]. Also, the spine has a natural shape at rest position. Theinitial joint position of each vertebra is input to the model.The range of movement of each region of the spine is di(cid:11)erent. For in-stance, the optimum movement of the lumbar region is (cid:13)exion or extension.The thoracic area easily moves laterally, while (cid:13)exion/extension in the sagittalplane is limited. The cervical area is very (cid:13)exible for both axial twisting andlateral bending. The joint limits for each region a(cid:11)ect how much that jointis able to participate in any given movement. The posture of the torso is aresult of the specialization of the spinal regions [Wil75].E(cid:11)ect of the Surrounding Ligaments and MusclesThe vertebrae are interconnected by a complex web of ligaments and muscles.If the force initiated by a muscle group is applied at one joint, the joint movesand the neighboring joints also move to a lesser degree. Some joints fartheraway might not be a(cid:11)ected by the initiator joint’s movement.It is possible to deactivate joints that are not initiating the movement.This action is achieved by simultaneous contractions of extensor and (cid:13)exormuscles around the spinal column [Wil75]. Depending on the force of theseresisting muscles, the joints on or near the joint closest to the resistor willmove less than they would if the resisting force had not been applied. The(cid:12)nal position of the spine is a function of the initiator force, the resistingmuscle, and the amount of resistance.2.3.2 Input ParametersThe spine is modeled as a black box with an initial state, input parameters,and an output state [MB91]. To initiate movement of the spine, several inputparameters are introduced. These parameters are:joint range FROM and TO:Within the total number of joints in thespine, any non-empty contiguous subset of vertebral joints may be spec-i(cid:12)ed by two joint indices. These joints indicate which part of the spineis active in movement. For example, the user speci(cid:12)es movement in therange between T5 and T10. All other joints are frozen in the movement.initiator joint:The joint where movement begins, usually the joint withgreatest motion.resistor joint:The joint that resists the movement. This may be equatedto a muscle that contracts and tries to keep part of the spine immobile.resistance:The amount of resistance provided by the resistor joint.38CHAPTER 2. BODY MODELINGspine target position:This is a 3D vector describing the target positionafter rotation around the,, andaxis. The target position is thexyzsum of all joint position vectors in the spine after movement succeeds.zero interpolation:A value of \yes" indicates that movement is interpo-lated through the joint rest position. A value of \no" indicates that onlythe joint limits are used to interpolate movement.2.3.3 Spine Target PositionThe joint between each vertebra has three degrees of rotation. The spine willmove toward the target position by rotating around the three possible axes[Lou83]:ROTATION OF THE SPINE(cid:13)exion/extension Forward/backward bending Rotation aroundaxisaxial rotationTwistingRotation aroundaxislateral bendingSide bendingRotation aroundaxisxyzThe position of the (cid:13)exion rotational axis for each vertebral joint hasbeen measured from cadavers, and is not equidistant to the two adjacentvertebrae, but is closer to the bottom vertebra [Lou83]. The origin of the axisof movement determines how the vertebrae move. When the torso is modeledon the spine, the axis also directly determines how the torso changes shape.Elongation and compression are absent from the model. The hydrophilicintervertebral disc, when submitted to prolonged compression induces a slightdecrease in height due to (cid:13)uid leakage. Conversely, after a long period of restor zero-gravity, the spine elongates by maximum (cid:12)lling of the nucleus pulposus(at the center of the disc) [Lou83]. Dehydration during a day’s activity canresult in a loss of height of 2 cm in an adult person. In any short durationof movement the disc is essentially incompressible, and therefore elongationis imperceptible [Hol81].Shearing or sliding (translational movements) of the vertebrae would leadto variation in the intervertebral separation. This would not be allowed bythe mechanics of the intervertebral disc [Lou83]. Therefore, the assumptionis made that for normal activities the three degrees of rotational movementare the only ones possible for each vertebral joint.2.3.4 Spine DatabaseAny human (cid:12)gure can have a wide variety of torso shapes. Also, each personhas a di(cid:11)erent degree of (cid:13)exibility and range of movement. In order to modelthe position and shape changes of an individual’s spine, a database has beendesigned for creating a unique set of features for the spine and torso. Medicaldata is the source of the database elements of an average person [Lou83].The database consists of the size of each vertebra in thedimension,x; y; z2.4. SHOULDER COMPLEX39Figure 2.1: Spherical Tra jectory of the Shoulder.the intervertebral disc size, the joint limits (3 rotations with 2 limits perrotation), and the joint rest (initial) position. In Section 4.2.3 we will see howspine movement is realized.2.4 Shoulder Complex5It is well known that the movement of the humerus (the upper arm) is not amatter of simple articulation as most computer graphics models would haveit. The movement is caused by articulations of several joints { glenohumeraljoint, claviscapular joint and sternoclavicular joint. Collectively, they arecalled the[EP87, ET89].shoulder complexIn, the shoulder complex is simpli(cid:12)ed with two joints { one connect-Jacking the sternum to the clavicle and the other connecting the clavicle to thehumerus [GQO89]. We call the former joint theand the latterclavicle joint+the. This simpli(cid:12)cation implies that the rotational center of theshoulder jointhumerus lies on a spatial sphere when the upper arm moves. It turns out thatit is very close to empirical data collected with a 6-D sensor attached to theexternal midpoint between the dorsal and ventral side of the right upper arm[Mau91]. The-axis of the sensor was along the longitudinal axis of the upperzarm with the positiveaxis direction pointing proximally, and the negativezy-axis pointing into the upper arm. The sensor was placed 10 inches from theshoulder (the extremal point of the humerus).From the experimental data the tra jectory of the shoulder can be easilycomputed. We (cid:12)tted the tra jectory by the sphere which yields minimumaverage residual error. The result is quite satisfactory: the radius of thesphere is 6.05cm, and the average error (the distance from the tra jectorypoint to the sphere) is 0.16cm. (Figure 2.1). Notice that the radius of the5Jianmin Zhao.40CHAPTER 2. BODY MODELINGFigure 2.2: A Neutral Human Figure.sphere is not the same as the dimension of the clavicle. This is due to thefact that the shoulder complex has indeed three joints instead of two. Thenet result can be modeled by two joints, however, if we put the center of theclavicle joint inbetween the two clavicle extremes.In our applications we feel that two joints are adequate to model theshoulder complex. So, we modeled the shoulder complex by grouping thesetwo joints into a joint group. Now we need to de(cid:12)ne the group angles and theway they drive the internal joint angles.2.4.1 Primitive Arm MotionsIt will be convenient to have the group angles of the shoulder complex describethe arm’s motion in a natural way. To focus on this motion, we assumethat the elbow, the wrist and all joints down to (cid:12)ngers are (cid:12)xed. What arethe arm’s primitive motions? Mathematically, any three independent arm’smotions will su(cid:14)ce. But careful selection will pay o(cid:11) in positioning ease.The arm’s motion can be decomposed into two parts: spherical and twist-ing motions. The motion which moves the vector from the proximal end tothe distal end of the upper arm is called spherical motion, and the motionwhich leaves this vector unchanged is called twisting.In a neutral body stance, consider a coordinate system where theaxiszpoints vertically downward, theaxis points towards the front, and thexyaxis points to the right.If we place the starting end of the vector fromthe proximal end to the distal end of the upper arm at the origin of thecoordinate system, the terminating end will stay on a sphere with the centerat the origin when the arm moves. The spherical motion can be furtherdecomposed into two motions: one which moves the arm vector along thelongitude of the sphere (elevation), and another which moves the vector along2.4. SHOULDER COMPLEX41the latitude (abduction). To describe the current status of the arm vector, weneed a convention for zero elevation or abduction. Let us de(cid:12)ne the amountof elevation as the unsigned anglebetweenaxis and the arm vector and,(cid:30)zfor the left arm, the amount of abduction as the signed anglebetween the(cid:18)(cid:0)yxyaxis and the pro jection of the arm vector on theplane. The positiveabduction is de(cid:12)ned when the absolute angle fromto the pro jection is(cid:0)y(cid:14)less than 180. Joint limits can be speci(cid:12)ed in terms of these spherical or\globographic" limiting angles [EP87].2.4.2 Allocation of Elevation and AbductionNaturally, the joint group shoulder complex will have elevation, abductionand twist as its group angles. They will be realized by internal joints | theclavicle and the shoulder joints. The amounts of elevation and abduction ofthe arm are allocated to the shoulder joint and the clavicle joint, while thetwist is allocated to the shoulder joint alone.According to clinical data, Otani gave a formula for distributing elevationand abduction to the shoulder and clavicle [Ota89]:(cid:30)c= cos()+ (1cos())90(2.2)(cid:18)(cid:12)1(cid:0)(cid:18)(cid:12)2 (cid:0)(cid:18)c= 02:(cid:18)(2.3)(cid:30)s=(cid:30)(cid:30)(cid:0)c(2.4)(cid:18)s=(cid:18)(cid:18)(cid:0)c(2.5)whereandare total elevation and abduction of the shoulder complex,(cid:30)(cid:18)subscripts \c" and \s" stand for the portions carried by the clavicle andshoulder joints, respectively, and(cid:12)1=(2.6)02514+ 91076 for 01314:(cid:30):(cid:20)(cid:20)(cid:30):(cid:26)(cid:0)0035+ 1287for1314:(cid:30):(cid:30) >:021066+ 92348 for 01300:(cid:30):(cid:20)(cid:20)(cid:30):(cid:12)2=:(2.7)(cid:26)1200:for1300(cid:30) >:2.4.3 Implementation of Shoulder ComplexIn,PeabodyRx; y; z()(28):is used to denote a generic rotation about the axis (). Since we needx; y; zrotations about some coordinate axis, letR!x()(29):represent the rotation aboutaxis bydegrees. Analogous notation will bex!used for theandaxes.yzPeabodyuses row vector convention. So a string of rotations, when readfrom the right, can be interpreted as applying the (cid:12)rst rotation (the rightmost42CHAPTER 2. BODY MODELINGone), then applying the next one about the rotated reference frame, and soon. When read from the left, it should be interpreted as applying successiverotations about the (cid:12)xed starting frame.With respect to the standing reference frame the spherical motion can bedescribed asR;;R;;;(100)(001)(cid:3)(210):that is, to achieve elevation and abduction of amountsand, the arm can(cid:30)(cid:18)rotate about theaxis by, followed by rotating about the unrotatedx(cid:30)yaxis by. This is just spherical coordinates which designate a point on a(cid:18)sphere. The (cid:12)rst problem we shall encounter is the singularity inherent to thespherical coordinate system.Dealing with Singularities of Spherical CoordinatesAs is known, the spherical coordinates have two singularities: when the eleva-(cid:14)tion is 0 or 180, it represents the pole (south pole for 0 or north pole for 180)no matter what the longitude (abduction amount) is. To see how this woulda(cid:11)ect the description of the spherical motion of the arm, let us do a smallexperiment. Starting with your left arm hanging down at your side, elevate(cid:14)(cid:14)(cid:14)your arm by 90, then abduct by 90, and (cid:12)nally elevate by90. Now see(cid:0)where your hand is. You will (cid:12)nd that your hand comes back but with anaxial twist. Where does this twist come from? It means that the pair of sode(cid:12)ned elevation and abduction motions are not independent from twist. Aslong as elevation is zero, the arm vector would not change. This is nothingbut a twist, as our decomposition of spherical motion and twisting motion in-tended. The (cid:12)nal coordinates are (0, 90), since the last elevation cancels the(cid:12)rst one and leaves the \abduction" there. To compensate for this unwantedtwist, we untwist the arm by the amount that the \abduction" would inducebefore the shoulder leaves the zero elevation. Therefore, we need a joint ofthree cascaded rotations as follows,R;;R;;R;;(cid:3)(cid:3)(001)(100)(001)(211):Let,andbe desired twist, elevation and abduction, respectively.(cid:28)(cid:30)(cid:18)Then the joint angles (or displacement, inlanguage) should bePeabody((cid:0))(cid:28)(cid:18); (cid:30); (cid:18):(212):The values in (2.12) are the amount of rotation about each of the correspond-ing axes in (2.11). The minusterm in the (cid:12)rst joint angle is to compensate(cid:18)for the unwanted twist induced by the abduction(so realized). As a matter(cid:18)of fact, the amount of twist is a relative quantity. It is meaningful only if wehave zero twist assumed for any direction of the arm. So (2.12) should be(+(cid:28)0(cid:28)(cid:18); (cid:30); (cid:18);(cid:0))(213):2.4. SHOULDER COMPLEX43whereis a function of (,), which de(cid:12)nes the zero twist for each (,).(cid:28)0(cid:30)(cid:18)(cid:30)(cid:18)Functionshould be such that(cid:28)0(cid:28)0; (cid:18);(0) = constant(214):since (0) denotes the same direction of the arm regardless of the value of; (cid:18)(cid:14)(cid:14). Now elevation by 90, followed by abduction of 90, and then elevation of(cid:18)(cid:14)(cid:0)90would wind up with((cid:28)0 (cid:0)90090);;:Since(0) = identity matrix, the (cid:12)nal matrixRxR(cid:28)z(0 (cid:0)90)(cid:3)R(cid:3)Rx(0)z(90)would be equal to(), which is exactly the starting con(cid:12)guration withoutR(cid:28)0ztwisting.Now let us take care of the other singular point, that is, when= 180.(cid:30)At this point,(180)= identity. But we haveRxRx(180)() =(cid:3)zz((cid:0)(cid:3))x(180)R(cid:18)R(cid:18)R:To deal with this singularity, we require that(cid:28)0; (cid:18)(cid:18)(180) = 2+ constant(215):to compensate for unwanted twist induced by \abduction". When= 180,(cid:18)(cid:30)R(cid:28); (cid:18)(cid:28)(cid:18)R(180) +(cid:0)(cid:3))z(0x(180)z()(cid:3)R(cid:18)(216):(cid:14)(cid:14)=((180) +))(180)(cid:0)(cid:3)(cid:0)(cid:3)z(xz0R(cid:28); (cid:18)(cid:28)(cid:18)R(cid:18)R=((180) +z0(cid:0)(cid:3)2)x(180)R(cid:28); (cid:18)(cid:28)(cid:18)R:(2.15) guarantees that that the (cid:12)nal con(cid:12)guration is independent of the ab-duction by.(cid:18)Combining (2.14) with (2.15), a possible choice ofcould be(cid:28)0(cid:28)(cid:30); (cid:18)(cid:18):0() =(217):(cid:30)90This is the choice in our current implementation. To achieve more natural zerotwist de(cid:12)nition, we need to (cid:12)ne tune. One possibility is shown in Section(cid:28)04.1.2 based on [Hut70] and [BOK80]. But (2.14) and (2.15) are required todeal with the singularity due to the spherical coordinate system.Dealing with Hierarchical Nature of ConnectionSo far, we have focused on the shoulder joint. However, as we argued, theshoulder complex will consist of two joints { shoulder and clavicle joints. Thetotal amount of elevation and abduction will be distributed to shoulder andclavicle joint according to the formula in Section 2.4.2.In thehuman (cid:12)gure model, the clavicle joint has the formJackR;;R;;;(100)(010)(cid:3)(218):644CHAPTER 2. BODY MODELING(cid:8)Hysterno-clavicle junction(cid:8)H(cid:8)H(cid:8)H(cid:8)HzHjShoulderx(cid:8)(cid:8)(cid:25)FrontsternumHYHHHHH(cid:8)(cid:8)(cid:8)Shoulder(cid:8)(cid:8)x(cid:8)(cid:8)(cid:25)yFront?z?Figure 2.3: Site Orientations at Clavicle and Shoulder Joints.Shoulder(cid:8)(cid:8)A(cid:8)(cid:8)A(cid:8)XXySternumX(cid:8)X(cid:8)(cid:3)XA(cid:8)X(cid:11)X(cid:8)AXX(cid:8)(cid:25)y(cid:8)X(cid:5)Elevation due tothe clavicle jointAAAAAzAU?z’Figure 2.4: Tilted Shoulder Frame.since the site orientation at the clavicle joint is a little bit di(cid:11)erent from that atthe shoulder joint (Figure 2.3). We do not need to worry about the singularityproblem here as we did with the shoulder joint because the arm, which theshoulder complex drives, is not aligned with the center of this joint. Theamount of elevation and abduction of the clavicle is given in (2.22.5). The(cid:24)clavicle is closer to the (normal) root of the human (cid:12)gure than the shoulder.This means that the global (relative to the neutral torso) orientation of thesite at the shoulder joint will be altered by the movement of the clavicle: thezaxis of the site at the shoulder would no longer point vertically downward.This causes the tra jectory of the hand drawn by \abduction" to no longerbe on the horizontal plane. Notice that theaxis will always be horizontal,xand hence the meaning of elevation is not challenged. To protect \abduction"against this alteration, we need to do a transformation of the site coordinates.After some movement of the clavicle, the shoulder reference frame is tiltedas shown in Figure 2.4. (The Figure displays only the plane= 0. Theaxisxx2.5. CLOTHING MODELS45does not show up in the Figure, but should be understood by the right-handprinciple.) The tilted abduction is due to the tiltedaxis. The \correct" ab-zduction should be the rotation about the dottedaxis. This axis is not con-stant due to variable clavicle elevation, but thelanguage requires aPeabodyconstant axis. To circumvent this restriction, we can perform a transformationof coordinates. The rotation axis described in () is simply(001).x; y; zR;;00The matrix representation of a rotation in a di(cid:11)erent coordinate system isjust a similarity transformation of the coordinate transformation matrix. Letvectors in () be primed. The transformation of coordinates isx; y; z00vR(cid:11)v=()x(219):0whereis the amount of elevation allocated to the clavicle joint. The rotation(cid:11)000aboutaxis byrepresented in () is() but, when representedz!x; y; zR!in the old coordinate frame (), it would bex; y; z0z0zR(cid:11)R!R(cid:11):xz()()x((cid:0))(220):Substituting this for the second(001) in the shoulder joint cascaded rota-R;;tions (2.11), we getR;;R;;R;;R;;R;;:(001)(100)(100)(001)(100)(cid:3)(cid:3)(cid:3)(cid:3)We do not need to substitute for the (cid:12)rst(001) in (2.11), because it isR;;there for twist and to compensate for unwanted twist caused by the abductionallocated to the shoulder joint. Two contiguous rotations about theaxisxcan be combined into one. The (cid:12)nal form of the shoulder joint becomesR;;R;;R;;R;;:(001)(100)(001)(100)(cid:3)(cid:3)(cid:3)(221):Letbe the amount of twist and the notations of elevation and abduction(cid:28)be as in Section 2.4.2. The joint angles for the clavicle take values(cc)(cid:18); (cid:30)(222):with respect to (2.18), and the joint angles for the shoulder take values(+ ((cid:28)(cid:0)1)(cid:18); (cid:30); (cid:18);(cid:30):ss(cid:0)c)(223):with regard to (2.21). It is not di(cid:14)cult to verify that the hand orientationdoes not depend on the abduction allocated to the shoulder joint,, when(cid:18)sthe total elevation,, is 0 or 180.(cid:30)(cid:30)90(cid:14)2.5 Clothing Models6In most workplace environments we have encountered, clothed (cid:12)gures are thenorm and would be expected by the designer. Adding clothing to a human6Eunyoung Koh.46CHAPTER 2. BODY MODELING(cid:12)gure improves its graphical appearance and realism. Clothes modeling can bedone in many ways ranging from very simple to more realistic but complicated.The simplest clothing technique is to change the attributes of certain segmentsof the body (cid:12)gure; for example, by modifying the colors of the pelvis and upperleg segments we get the e(cid:11)ect of a body wearing short pants. This is not quiteas silly as it sounds, because the body segment geometry can be created witha clothed rather than bare-skinned shape. The best but more complicatedapproach is to drape and attach clothing over a body to simulate the intricateproperties of garments.Besides improving realism, there is a practical human factors aspect toclothing. Clothing constrains movement by restricting the joint angle limits.Preliminary attempts to analyze this problem use collision detection over ageometric clothes model.2.5.1 Geometric Modeling of ClothesRigid clothing models are created by designing special segment psurfs. Thusa shirt or jacket would have, say, (cid:12)ve parts: one for the torso, and two foreach limb segment. Clothing exists independently of a given (cid:12)gure modelas a library of ob jects which can be selectively placed on a model at userdetermined sites. This database is modi(cid:12)able through typical geometric edit-ing commands. A clothing item is positioned by matching key points on theclothing to key points on the body segments. A global deformation algorithm[SP86] can be used to (cid:12)t the clothing piece correctly on the segment.One apparent problem with geometrically modeled clothing occurs whenthe human (cid:12)gure moves joints. Since the clothing model is not deformable,there are gaps between segments. (This is in fact true even without clothingif the (cid:12)gure is modeled with polyhedral meshes. As the geometry is carriedon the segment, it inherits the geometric transformation without any com-pensation for the interaction of material, (cid:13)esh or clothes, at the joint.) Extrawork is necessary to alleviate the joint gap problem. A gap (cid:12)lling algorithmhas been developed to make up these gaps when animating. It connects theboundaries of two adjacent segments by generating spline surfaces using thetangent information of the two segments at the boundaries.As an initial attempt to develop geometric models of clothes, a sweatshirtand pants were designed (Plate 2). Each segment of a clothing item is a psurfwhose geometry and position are closely related to a corresponding bodysegment. The following is a step by step procedure for geometric clothesdesign.2.5. CLOTHING MODELS471. Determine the size of clothes:In conventional clothing design, circumferences are measured at certainpositions of the body in order to determine the clothing size. In ourapproach, the maximum segment breadth in two orthogonal directionsare measured instead of circumferences.The following list shows the positions at which the maximum breadthsare measured and lists their corresponding slices from the accuratebiostereometric bodies (Section 2.1.3).(cid:15)For trouserswaistthe 6th slice of lower torsohipthe 9th slice of hip (cid:13)apupper legthe (cid:12)rst slice of upper leglower legthe last (24th) slice of lower leg(cid:15)For a sweatshirtneckthe (cid:12)rst slice of upper torsochestthe 6th slice of upper torsobreastthe 12th slice of upper torsowaistthe (cid:12)rst slice of lower torsoupper armthe 8th slice of upper armlower armthe last (16th) slice of lower armAlso, a measure of the length from the neck to the bottom of the shirt isprovided to determine the length of the shirt. These sizes are speci(cid:12)edin advance for each article of clothing.2. Create psurfs:The geometry of each segment is determined by a set of body slices.A new set of slices for clothing is constructed by sculpturing the bodyslices depending on the shape of the segment and the speci(cid:12)ed sizes inthe previous step. The fundamental idea in the construction is to pick afew thick slices and duplicate them appropriately along the segment afterscaling. Scaling is done by linear interpolation so that the scaled slicesmay match with the speci(cid:12)ed maximum breadth sizes at the positionsdesignated.The completed surface de(cid:12)nition of a clothes segment can be obtainedby tiling the slices. Tiling is performed by generating rectangles fromthe data points which de(cid:12)ne two adjacent slices.3. Attach clothes segments to human body:Each clothes segment can be attached to the corresponding body seg-ment by a joint which is located at the upper part of that segment.48CHAPTER 2. BODY MODELINGThe clothing shape can be easily modi(cid:12)ed by changing the slice de(cid:12)nitionof the clothes. For example, folded sleeves, short sleeves, and short pants canbe simulated by simple modi(cid:12)cation or deletion of slices.2.5.2 Draping ModelThe most realistic clothing can be created by simulating the support anddraping of pattern pieces of arbitrary shape. Wrinkling, folding, and the ef-fects of gravity are displayed through a relaxation method or a (cid:12)nite elementmethod. Pattern pieces may also be stitched at seams and draped simultane-ously. Pattern pieces of di(cid:11)erent lengths may be sewn together, resulting inan oversewing e(cid:11)ect.The draping of the pattern pieces is done on a (cid:12)gure in a static posture.Interference testing is done in the draping algorithm to make sure that thepattern pieces slide over the surface of the (cid:12)gure without penetrating thesurface (Plate 3).There are several methods to simulate the draping of a square piece ofcloth, isolated from other cloth, which are based on a relaxation method.Feynman [Fey86] uses a formula which minimizes the energy of a cloth andtries to simulate the shape of thin (cid:13)exible membranes under the in(cid:13)uence offorce (cid:12)elds and rigid bodies. The local minimum of the cloth is found bymoving each of the grid points in turn toward a position which decreases theenergy of the cloth. The energy expression of a cloth is described as:ESksSkbSkgStotalsbg() =()()()(cid:0)(cid:0)where()()() represent the e(cid:11)ects of strain, bending, and gravity.sS; bS; gSThe parameterscontrol the relative strengths of these three e(cid:11)ects:k; k; ksbga largemeans the cloth is di(cid:14)cult to stretch; a largemeans the cloth iskskbsti(cid:11) and resists bending; and a largemeans the cloth is heavy.kgRelaxing a single point is the process of moving it so that the energy ofthe cloth of which it is a part is decreased. The method used to relax a singlepoint (cid:12)rst (cid:12)nds the direction in which the point would most like to move: thedirection of the negative gradient of the energy as a function of position. Thenit moves the single point in that direction so that its energy is minimized.Feynman suggests using a multigrid method to speed up the relaxationsweeping process. However, it must be used carefully to avoid distortion. Healso introduces (cid:12)xed points in order to forbid the cloth to move into a solid.Weil [Wei86] considered the problem of hanging a piece of cloth by (cid:12)xingsome constraint locations of the cloth. The cloth is represented as a rectangu-lar grid () of 3D coordinates (). His method is a two phase algorithm.u; vx; y; zThe (cid:12)rst part approximates the surface within the convex hull in () spaceu; vof the constraint points; that is, all the interior points are placed on catenaries.The second phase uses an iterative relaxation process to minimize maximumdisplacement of all the points in the grid up to a given tolerance.Terzopoulos, Platt, Barr and Fleisher [TPBF87] use elasticity theory todescribe the behavior of a deformable ob ject. The model responds in a natural2.6. THE ANTHROPOMETRY DATABASE49way to applied forces, constraints, and impenetrable obstacles. The equationsof motion governing the dynamics of the deformable bodies under the in(cid:13)uenceof applied forces is given by@@ r@ r(cid:14)Er()() ++=()(cid:22)(cid:13)fr; t;@ t@ t@ t(cid:14)rwhere() is the position of the particleat time,() is the massra; tat(cid:22)adensity of the body at,() is the damping density, and() representsa(cid:13)afr; tthe net externally applied forces.() is a functional which measures the netErinstantaneous potential energy of the elastic deformation of the body.To create animation with this model, the motion equation is solved nu-merically, integrating through time. This model is active in the sense that itresponds to forces and interacts with ob jects.2.6 The Anthropometry Database7While animation research may be content with demonstrating action on aconvincing human form, there is often only a single carefully structured (cid:12)gureinvolved. Its body dimensions may be estimated or obtained by measurementof a speci(cid:12)c individual. In contrast, engineering human factors applicationshave long been concerned with construction of valid ranges of human formsbased on empirically measured populations such as aircraft pilots, (cid:13)ight at-tendants, or astronaut trainees. These engineering applications recognizedthe need for a variety of accurately scaled body dimensions to facilitate reachand (cid:12)t analysis [Fet82, HBD80, KSC81, Doo82]. Unfortunately, most of thesesystems are either proprietary, hard-wired to some particular population, non-interactive, or otherwise di(cid:14)cult to use with contemporary graphical systems.Jack, however, permits an open and accessible database for human dimen-sional data. Interactive access is provided through a novel spreadsheet-likeinterface.2.6.1 Anthropometry IssuesAnthropometry, the science of human body measurement, has been an areaof interest throughout history [LRM88]:In his authoritative book \A History of the Study of HumanGrowth," Professor Tanner writes that the ancient Greeks, as wellas sculptors and painters of the Renaissance, measured the humanbody to estimate body proportions and, thus, reproduce life{likeimages of varying sizes. Interest in absolute size developed laterin the 17and 18centuries out of military concerns. The Euro-ththpean armies preferred taller soldiers, and recruiting o(cid:14)cers becameanthropometrists. Interest in scienti(cid:12)c study of growth and in the7Marc Grosso, Susanna Wei.50CHAPTER 2. BODY MODELINGrelative importance of nature versus nurture in explaining humanvariability has been pronounced since the 19century.thThe vast ma jority of work in \modern" anthropometry has been done byanthropologists who were studying the e(cid:11)ects of some environmental factoron some population. While there are studies dating back to the mid{ to late{1800’s, more recent studies covering groups of adults (i.e. populations) fromaround the world are summarized in the[NAS78].Anthropometry Source BookIts two volumes have become one of the foundation sources for contemporaryanthropometry.Anthropometric studies di(cid:11)er greatly in the number and kind of measure-ments selected. They all report a statistical analysis of the values of eachmeasurement, giving at least a median with standard deviation and the max-imum and minimum values. The studies typically report the above valuesalong with intermediate values at selected percentiles of the population, typ-ically 1, 5, 25, 50, 75, 95and 99, since body size data does notstththththththvary linearly with percentile.Some of the data found in these studies was used in theNASA Man{Systems Integration Manual[NAS87], as the basis for the estimated mea-surements for male and female astronauts in the year 2000, using the bodydimensions of American males and Japanese females.It is felt that thesepopulations provide the maximum range in body sizes in the developed worldtoday since the American male is among the largest of males and the Japanesefemale is the smallest of females. There is a growth rate factor which is usedto adjust the values for pro jection to the year 2000.The measurements selected for inclusion in theNASA Man{Systems In-tegration Manualwere chosen to meet the various needs of NASA and werenot intended to be a complete set of measurements for all purposes or for allpossible users. These measurements were publicly available, however, and de-tailed enough to satisfy many ergonomic analysis requirements. They servedas the basis for the human (cid:12)gure model we developed but are are not completeenough to totally describe it [GQO89, GQB89]. Some needed measurements+and data are missing; though most of the missing values can be found in theAnthropometry Source Book, there were a number of measurements requiredfor our model which were not easy to track down. Where this occurred, intel-ligent estimates have been made based upon data values from closely relatedmeasurements (possibly from a di(cid:11)erent population) or by calculating thevalues from other measurements. In no case were the unde(cid:12)ned values setarbitrarily.2.6.2 Implementation of Anthropometric ScalingEach body segment or structure having associated geometry contains length,width, and depth (or thickness) attributes. Therefore, we require a minimum88realistic animation.We presently ignore segment shape changes though we realize their importance for2.6. THE ANTHROPOMETRY DATABASE51of seventy{two (72) measurements to describe the physical dimensions of ourhuman (cid:12)gure.Psurfs describe the shape of each segment. Anthropometric scaling modi-(cid:12)es the segment dimensions as well as any associated psurfs. It is very simpleto change to alternative polygon models, e.g. to the detailed contour bod-ies, to vary detail in the segment while preserving the correct anthropometricscale. Each psurf for the various segments is stored in a normalized formatwhere the(length) dimension ranges from 0 to +1, and the(depth) andzxy(cid:0)(width) dimensions range from1 to +1. In order to display these psurfs,using either real measurements for a person or percentile measurements forsome speci(cid:12)ed population, the psurfs must be scaled.Body de(cid:12)nition (cid:12)les containing the desired values can be created (or modi-(cid:12)ed) by manually entering the body part names and their values in the properformat, but this is clumsy and rarely used. A superior approach uses theSpreadsheet Anthropometry Scaling System () which will be discussedSASSin detail in Section 2.7.2.6.3 Joints and Joint LimitsThere are three di(cid:11)erent types of human body joints [TA75]: Fibrous, Car-tilaginous, and Synovial. Of these three we are only concerned with thesynovial joints (joints with joint cavities). The synovial joints are categorizedbased upon the shape of the articulating surface of the joint. There are sevensub{types of synovial joints found in the human body [MB77, TA75]. Thesesubtypes are:(cid:15)Monaxial (or uni{axial) joints (1 DOF)a. Hinge joints. A convex surface of one bone (cid:12)ts in a concave surfaceof another bone. This joint allows movement in only one plane,usually extension and (cid:13)exion, similar to that of a door hinge. Ex-amples are the elbow joint, knee joint, ankle joint, and interpha-langeal joints (joints in the toes and (cid:12)ngers).b. Pivot joint. A rounded, pointed, or conical surface of one bone artic-ulates with a shallow depression in another bone. The primary mo-tion of this joint sub{type is rotation. Examples are shown by thesupination and pronation of the palms, atlas{axis joint (Alanto{Axial joints located at the very top of the spine), and radioulnarjoint (between radius and ulna in forearm).(cid:15)Bi{axial joints (2 DOFs)a. Condyloid joints. These are the joints like those at the heads ofthe metacarpals (hand bones), i.e. the knuckles, which is the bestexample of this type of joint.52CHAPTER 2. BODY MODELINGb. Ellipsoidal joints. The oval{shaped condyle (end) of one bone (cid:12)tsinto the elliptical cavity of another bone. This type of joint permitsside{to{side and back{and{forth movements (in the principal axesof the ellipse). Examples are shown by the (cid:13)exion and extensionand abduction and adduction of the wrist (radiocarpal) joint.(cid:15)Tri{axial (or multi{axial) joints (3 DOFs)a. Saddle joint. Both bones in this joint are saddle{shaped, thatis convex in one direction and concave in the other. This type ofjoint is essentially a modi(cid:12)ed ellipsoidal joint and has more freedomof movement. Saddle joints allow side{to{side and back{and{forthmovements as well as rotation. An example is the joint between thetrapezium and metacarpal bones of the thumb (carpometacarpaljoint of the thumb).b. Ball and socket joints. A ball{like surface of one bone (cid:12)ts into acup{like depression of another bone. These joints permit (cid:13)exion{extension, abduction{adduction, and rotation. Examples are thehip and shoulder joints.c. Gliding (or plane) joints. Bones involved have (cid:13)at or nearly (cid:13)atarticulating surfaces. Movement can occur in almost any plane,with side{to{side and back{and{forth movements the most com-mon. The movements are always slight. Examples of this type ofjoint can be found between the carpal (wrist) bones (intercarpaljoints), between the tarsal bones (foot/ankle) (intertarsal joints),between the sacrum (lower end of the spine) and ilium (a hip bone)(the sacro{iliac joint), between the sternum (breast bone) and clav-icle (collar bone), between the scapula (shoulder blade) and clav-icle, between the individual vertebral arches, at the heads and atthe tubercles of the ribs, and at the front ends of the costal (rib)cartilages.Each joint in the human body has a range of motion over which it will al-low movement to occur. A joint’s range of motion is determined by a numberof factors including joint type, muscle size at the joint, muscle tension (tonus)for the muscles at the joint (i.e. (cid:12)tness of the person), ligament stretchabil-ity or give, amount of fatigue, and training adaptations for the joint. Theterm (cid:13)exibility is frequently used to describe the in(cid:13)uence that each of thecomponents listed above has on joint movement.Joint range of motion, described in terms of angles, is measured in degreesfor each DOF, that is, each plane in which movement is allowed at a joint.When a joint has more than one DOF, the range of motion at the joint for eachDOF may be variable because one DOF may in(cid:13)uence the others. Also, forjoints which are in(cid:13)uenced by muscles crossing two joints (as in some musclesof the thigh, for example) there may be a two joint dependency on the jointlimit.2.6. THE ANTHROPOMETRY DATABASE53Jackincorporates upper and lower joint limits for every single DOF joint.For two DOF joints, independent limits for each DOF are used. But theshoulder is treated as a three DOF system with spherical joint limits anda function that relates the default upper arm orientation to the upper armposition (Section 2.4).respects these joint limits during both interactiveJackpositioning and inverse kinematic reaching.2.6.4 MassAs dynamic simulations achieve ever more realistic animation, mass informa-tion becomes essential. Fortunately, along with stature, mass is among themost common body measures taken. There have been a number of studieswhich have determined that each of the various body segments contributesa certain percentage of the total body mass; this percentage determines themass of each individual segment. The mass percentages used are averagepercentile values for a (cid:12)t male population as would be found in the NASAmale crewmember trainees. The distribution may very well di(cid:11)er for theaverage general population or a population which is skewed toward eitherthe small/light weight (like horse racing jockeys) or large/heavy weight (likeAmerican Football lineman). The segment mass percentages are also likely tobe di(cid:11)erent for female sub jects.2.6.5 Moment of InertiaThe concept of moment of inertia is important when attempting to describethe dynamic behavior of a human (cid:12)gure. These values are needed when de-termining the motion of a (cid:12)gure under the in(cid:13)uence of forces (both externaland internal), moments, and instantaneous transfers of momentum (i.e. col-lisions). When considering human (cid:12)gure modeling the common forces andmoments e(cid:11)ecting the human (cid:12)gure include:1. gravity: a force acting at the center of mass of each segment with amagnitude proportional to the segment’s mass.2. internal forces generated by muscles:forces actually acting at someinsertion point along the length of the segment but modeled as a drivingmoment applied at the joint.3. reaction forces generated by the (cid:12)gure’s surroundings: for example, thenormal forces and friction forces applied to the (cid:12)gure’s hand by thesurface it is leaning on.attempts to pull, etc.4. external forces: for example, weights lifted by the (cid:12)gure, levers the (cid:12)gure5. collisions: usually approximated as an instantaneous change in velocityof the point on the (cid:12)gure being struck.54CHAPTER 2. BODY MODELING2.6.6 StrengthHuman strength capabilities have been found crucial for more realistic andnatural human motions (Section 5.3). Human strength (maximum torques) isde(cid:12)ned as muscle group strengths and is stored on a joint DOF basis. Mod-eling strength this way allows di(cid:11)erent people to possess di(cid:11)erent capacitiesin di(cid:11)erent muscle groups. Thus, the strength di(cid:11)erences between two people{ such as a dancer and a pianist { can be readily modeled and illustrated.Each DOF of a joint has two joint movements which are associated with twodi(cid:11)erent muscle groups. For example, an elbow joint is modeled to have oneDOF because it can only rotate around one axis. Its rotational movements are(cid:13)exion and extension, corresponding to the (cid:13)exor and extensor muscle groups.Therefore, strength data of (cid:13)exion and extension are stored for an elbow joint.Each muscle group strength is modeled as a function of body position, anthro-pometry, gender, handedness, fatigue, and other strength parameters [AHN62,Lau76, NAS78, AGR81, AGR82, Imr83, CA84, HJER86, MS86, NAS87].++In terms of body position, we choose a more generalized model that takes thee(cid:11)ects of adjacent joint angles into consideration [Sch72]. For example, themuscle group strengths of a shoulder joint are modeled to be functions notonly of the shoulder angles but also of the elbow angle.2.7 The Anthropometry Spreadsheet9Given the large number of data items needed for anthropometric body sizing,a spreadsheet{like format was a natural choice for the user interface. We calledit: the Spreadsheet Anthropometry Scaling System.SASSSASSwas originally developed with one idea in mind, i.e., generatingthe dimensions of each segment of a human (cid:12)gure based upon populationsupplied as input. The human model used by the current version of SASSconsists of thirty-one segments (body structures), of which twenty-four havea geometrical representation. For each of those twenty-four segments, thereare three dimensions which are required, namely, length, width, and thickness.This means that at least these seventy-two measurements should be available.The psurf geometry of each segment must be scaled by real measurementsfor a person, or percentile measurements for some speci(cid:12)able population.SASSgenerates (cid:12)gure (cid:12)les with the appropriate description of the segmentdimensions and joint limits, so thatcan display the resulting (cid:12)gure.JackSASSuses population statistic data to create generic human (cid:12)gures. Al-ternately,has a built-in database that stores anthropometric data forSASS(real) individuals and provides an interactive query system for database ac-cess.9SASSallows (cid:13)exible interactive access to all variables needed to size ahuman (cid:12)gure described structurally by abody (cid:12)le. ThePeabodySASSscreens, as shown in Fig. 2.5 and more diagrammatically in Fig. 2.6, areRichard Quach, Francisco Azuola, Welton Becket, Susanna Wei.2.7. THE ANTHROPOMETRY SPREADSHEET55Figure 2.5: Sample Display From SASS.divided into di(cid:11)erent sections including anthropometric group selection, globaldata, command menu, local data.Data that may be accessed is organized into anthropometric \groups". Thecurrent version can handle four groups: segment girth, joint limits, segmentcenter of mass, and strength.The global data section of the spreadsheet is intended to allow a \wholebody" view of the current (cid:12)gure parameters. Currently, the six items con-sidered for any human (cid:12)gure are:population, sex, (cid:12)gure type, mass,statureoverall percentileSASS, and. It is important to realize that sinceis a relational spreadsheet, modifying any data in this section will a(cid:11)ect thevalues of the individual segments. For example, changing the (cid:12)gure’s per-centile will cause the data to be scaled in other appropriate segments con-tributing to stature.The data section is used for the display of individual segment data andtheir corresponding percentiles. The leftmost column is reserved for the seg-ment names, while the other six columns are used for the data and percentiledisplay. The segment name column cannot be modi(cid:12)ed. The data is read infrom the selected population input (cid:12)le.56CHAPTER 2. BODY MODELINGAnthropometric GroupGlobal data Command MenuLocal Data SectionCommand/Message lineFigure 2.6: SASS Screen Layout.Data and its corresponding percentile is modi(cid:12)ed by simply moving thelocator device to the desired cell and pressing on a button. Changing anysegment percentile will change its corresponding dimension.keeps aSASScurrent measurement unit type for each group (). Unitin, cm, deg, radconversion is performed when necessary.2.7.1 Interactive Access Anthropometric DatabaseAn anthropometric database stores attribute data for a speci(cid:12)c set of individ-uals rather than a population. Each person has associated groups of anthro-pometric data: girth (segment dimensions), joint limit, center of mass, andstrength. Each group of anthropometric data is stored in a separate relation.In the application of a task simulation, it is very important to (cid:12)nd anindividual with the requisite anthropometric characteristics such as body di-mensions, joint limits, and strength.provides a query system usingSASSpop-up menus to allow the user to select the people with the desired charac-teristics from the anthropometric database. Therefore, the user does not needto know the database structure nor its query language.The user can query on di(cid:11)erent anthropometric characteristics separatelyor in combination using operationsand, or, greater than, equal to, less than,etc. For example, the user can inquire about people with the desired strengthcapabilities alone, or query about individuals with the required strength ca-pabilities, body dimensions, joint limits, and center of masses together. Theindividuals that satisfy the query and their global data are stored in a listcalled the. After examining the global information in the query list,query listthe user can choose all or some of these individuals and store them in theselected list. The detailed anthropometric data of each individual in the se-lected list can be displayed on the anthropometric spreadsheet. If desired, theuser can also create thestructure (cid:12)les for those selected individualsPeabodyby using thecommand.SASSCreate Figure2.7. THE ANTHROPOMETRY SPREADSHEET572.7.2 SASS and the Body HierarchyOur (cid:12)rst version ofhandled segments by gathering them in a simpleSASSlist. This was good enough to represent any number of segments, but pre-sented some inconveniences in de(cid:12)ning relations among the segments. In asubstantial modi(cid:12)cation, the structure was changed to a hierarchical tree. Atthe bottom of the tree the leaves correspond to the segments. The internalnodes correspond to conceptual body parts composed of sets of actual bodysegments, for example, the \arm" consists of the upper arm, lower arm, andhand segments. Thus concepts such as the \length of the arm" can be well-de(cid:12)ned. A (cid:12)gure can be de(cid:12)ned as a collection of body parts, joined togetherby joints. Each body part, in turn, can be de(cid:12)ned as a collection of bodysegments connected by joints.2.7.3 The Rule System for Segment ScalingThe introduction of the body part hierarchy permitsto determine andSASSuse attributes of body parts as well as individual segments. For example,SASSde(cid:12)nes a rule for computing the height of an individual as the sum ofthe segments’ lengths in a path that goes from head to feet. For those segmentsin the path, the rule allows varying their lengths if the stature changes and,vice versa, to change the stature if the length of any of the segments in the pathchanges. There is an alternate rule that keeps the stature (cid:12)xed and adjusts thesegments’ lengths accordingly, if the length of one of them varies. Another ruleincludes changing the mass according to the stature and, conversely, changingthe stature according to a speci(cid:12)c mass value.The underlying criterion for doing the stature changes is a linear one. Thesegments in the stature path are head, neck, upper torso, center torso, lowertorso, upper leg, lower leg, and feet. The length of each of these segments,except for the feet, is computed as the length in thecoordinate. For thezfeet, the length is thecoordinate (since for the feet, thecoordinate is theyzlongitudinal dimension). The thickness and width of the segments are nota(cid:11)ected by these changes, for there is no rule to decide the e(cid:11)ects of staturechanges in these parameters. The updating process must be done carefully,for it might happen that modifying the length of a given segment violatesthe range of possible stature values admitted by the current population orconversely, if the stature is changed, it might not be satis(cid:12)able by variationsin the segment lengths.The other case considers (cid:12)xed stature. The idea is to adjust the segments’lengths along the stature path if the length of one of them varies, such that theglobal length (stature) remains constant. While this might appear easy to doat (cid:12)rst, it is not a trivial matter. Recall that segment dimensions are based onpopulation percentile data suitably interpolated and are therefore restrictedto legitimate ranges. Furthermore, the stature itself is restricted by a \real"set of values (for each of the percentiles). When the user speci(cid:12)es a particularchange in the length (or other dimension) of a given segment, the underlying58CHAPTER 2. BODY MODELINGrule attempts to satisfy the constraint of (cid:12)xed stature, that is, it tries to keepthe stature value constant. For example, assume the length of the head hasdecreased. To keep the stature (cid:12)xed, the lengths of the other segments inthe stature path must vary in the opposite way. Currently, the modi(cid:12)cationis done in a linear fashion since there are no rules to de(cid:12)ne this otherwise.But it might be the case that in the updating process one of the segment’sdimensions (length) cannot be satis(cid:12)ed, that is, the resulting dimension is outof the range established by the 595percentile values. In this situation,(cid:0)ththththe rule sets the length to its closest limit (5or 95percentile value) andtries to satisfy the requirement of (cid:12)xed stature by modifying the remainingsegments in the path. Notice that there is a possibility that the stature cannotbe kept constant. There is one more step involved in the updating processfor (cid:12)xed stature:if the stature is varied by the user the segments changecorrespondingly if possible.It is important to understand the back and forth process that goes onbetween body parts and segments. If the overall body is supposed to be 50thpercentile, the body parts need not all be 50percentile. In fact, we do notthhave a rule yet to specify the percentile of the body parts (segment-wise) for agiven global body percentile. That information must come from multi-variateanalysis of population data. So we must be able to change dimensions of thebody parts or segments to comply with all the possiblecompositions ofvalida 50percentile body. If the stature is modi(cid:12)ed, then a new global percentileis computed. For that new global percentile, we have a speci(cid:12)c rule tellingus what the possible compositions are. Unfortunately, the compositions arenot unique since they depend on the population data used. To illustrate this,suppose we have the following (partial) composition set percentiles: feet 30%,legs 45%, torso 60%, head 40%,... for a 50percentile body. Then supposethwe want to change the stature in such a way that the resulting body percentileis 60percentile, and the analogous (partial) composition set is (feet 40%,ththlegs 56%, torso 50%, head 40%,...). Then we scale the ob jects in the staturepath (which are those listed in the composition sets) to comply with thissecond composition set. But we must be sure that there is no con(cid:13)ict indoing so; for instance, the feet might be able only to grow from 30% to a40% under the given population. In general, di(cid:11)erent populations will havedi(cid:11)erent compositions.In this example, the compositions were stated at the body part level. Theremust be an equivalent composition at the segment level: the segment versionof the composition for the 50percentile (cid:12)gure is, for instance, upper legth(45%, lower leg 60%) assuming legs decompose in two pieces. But what if thecompositions, even though being based on a particular population data, arenot available for all the possible percentiles? We would have to use a (cid:12)xedcomposition or else interpolate compositions (if it is sound to do that) andmake sure a given segment’s length is not violated (according to its percentilerange) when trying to go from a composition for the 50percentile (cid:12)gure toththat of the 60.thAnthropometrists who argue against the cavalier usage of percentiles in2.7. THE ANTHROPOMETRY SPREADSHEET59characterizing body size distributions would seem to be on pretty safe ground.Since the only obvious alternative involving enumerating all the individualsin a database we are stuck with the population statistics available. As multi-variate population dimension distribution data becomes available,willSASSbe ready to utilize it for proper (cid:12)gure creation.2.7.4 Figure CreationSASSJackcan produce a (cid:12)le containing the scaling of a (cid:12)gure whichtheninterprets to create the (cid:12)gure (cid:12)le. Scaling (cid:12)les o(cid:11)er a particular bene(cid:12)t overdirect creation of the (cid:12)gure in. Consider the situation in which theSASSJackuser wants to determine the percentile ranges of a (cid:12)gure that can satisfya given task, that is, the problem of (cid:12)nding the speci(cid:12)c (cid:12)gure percentile (orrange) that can (cid:12)t in a particular workplace. One can attempt to read eachof the possible (cid:12)gure (cid:12)les out oflibraries and try to keep the (cid:12)gure inJackthe desired position. The other, more sensible, option is not to load di(cid:11)erent(cid:12)gure (cid:12)les, but instead, to load di(cid:11)erent scaling (cid:12)les. Then the same (cid:12)gurecan be scaled using all these di(cid:11)erent (cid:12)les to (cid:12)nd the one that best suits thegiven environment. This is faster and more appealing to the user.2.7.5 Figure ScalingThe scales in the (cid:12)gure scaling (cid:12)le are obtained directly from the dimensions ofthe body segments or parts. Thus, this scaling (cid:12)le represents the dimensionsspeci(cid:12)ed in the (population’s) girth (cid:12)le, for a given (cid:12)gure percentile. If therewere only one scale factor for each dimension of a segment, there would besome scaling mismatches towards the ends of the segments. This is in facta ma jor problem, especially for the low resolution polyhedral body model.For instance, the scaled upper leg appears to be too thick and too wide, incomparison to the lower leg. The same problem occurs with the upper arm andthe lower arm. The scaling of the pelvis of the simple polyhedral body seemsto be too wide and thick, while the torso appears to be too narrow and short.There are various solutions to these problems. The simplest one is to adapt thedata to the model by modifying the scaling factors to obtain a good (looking)(cid:12)gure scaling. There are no strict rules for this, though. The rule we use is toconsider body lines as second order continuous curves. There are no abruptchanges from one body part to the next one (assuming no deformations). Thuswe approximate (in a rather arbitrary way) the scaling factors to achieve thiscontinuity. The largest discrepancies are the ones mentioned above. Otherminor ones are the scaled neck being too wide and hands being too narrow. Ingeneral, the scaling factors are not changed by more than 10% in the simplepolyhedral(cid:12)gure.JackThe scaling factors generated byare mapped into the contour bodySASSwith almost no modi(cid:12)cations necessary (for the contour (cid:12)gure case, adjust-ments are done to the torso, legs, arms, and hands). These adjustments donot go over 5% of the actual values. Again, one must keep in mind that even60CHAPTER 2. BODY MODELINGthough the contour model is a more accurate representation of the humanbody, it is not a perfect one. Moreover, we must remember that theSASSscaling factors (cid:12)le is created based on a given population and the (cid:12)gure re-sulting from that scaling might not completely match a real human being (forsuppose that the population’s average torso length is greater than the torsolength of a given individual and the population’s average leg length is smallerthan that of the same individual, then we end up with a not so real scaling forthe contour model). Thus, even though we have assumed some adjustmentsare required, it is still necessary to prove if this is the right way to proceed. Sofar, the criterion that prevails is to display a good-looking (well proportioned)human (cid:12)gure.2.8 Strength and Torque Display10Human strength information has applications in a variety of (cid:12)elds. In phys-ical education, it can be used to classify participants for speci(cid:12)c activities orto search for a body position or motion which will improve an athlete’s perfor-mance; in medicine, it can be used as an indicator for a muscular injury or dis-ease; and in ergonomics and human factors, it can be used to guide the designof workspace or equipment to reduce work related injuries, and to help in per-sonnel selections to increase work e(cid:14)ciency [MKK88, McD89, Eva88, CA84].+Human strength information can also be used in a human task simulation en-vironment to de(cid:12)ne the load or exertion capabilities of each agent and, hence,decide whether a given task can be completed in a task simulation [EBJ89].To convey the multi-dimensional characteristics of strength data, we usespecial data display methods that use human (cid:12)gures together with two orthree dimensional graphics. Various forms ofandstrength boxstrength bardisplays allow visualization of strength parameters, give a dynamic changingview of the e(cid:11)ects of parameters on strength, and show safe and forbiddenregions in terms of di(cid:11)erent strength capabilities of individuals or populations[WB92].We de(cid:12)ne strength (maximum torque) to be the maximum achievable jointtorque. It is modeled as muscle group strength and is stored on a joint DOFbasis in the human (cid:12)gure model. Each DOF of a joint has two joint move-ments which are associated with two di(cid:11)erent muscle groups. For example,an elbow is modeled to have one DOF. It can only rotate around one axis;its rotational movements are (cid:13)exion and extension which correspond to the(cid:13)exor and extensor muscle groups. Therefore, strength information of (cid:13)exionand extension is stored for an elbow joint. Each muscle group strength ismodeled as a function of body position, anthropometry, gender, handedness,fatigue, and other strength parameters [AHN62, AGR81, NAS87, HJER86,+Imr83, Lau76, NAS78].10Susanna Wei2.8. STRENGTH AND TORQUE DISPLAY612.8.1 Goals of Strength Data DisplayStrength depends on a set of parameters. To properly display these parame-ters, one should use a multi-parameter system that can convey the meaningof these parameters as well as their values. Unfortunately, existing multi-dimensional display methods [And60, Ber83, GFS71, Har75, And72, Che73,KH81] are unsuitable in presenting the human body-speci(cid:12)c semantics ofstrength parameters. They are also visually ine(cid:11)ective in displaying strengthdata because they fail to tie the parameters to the body properties they de-scribe. We prefer to de(cid:12)ne multi-dimensional graphing methods which usethe body itself as a context for the parameter display.Strength data displays show the current force and torque capabilities of a(cid:12)gure in a given posture or motion. A good strength data display should showthe direction and magnitude of forces and torques. Since strength dependson many parameters, for example, body posture, body size, gender, fatigue,training, and handedness, it is useful to observe interactively how the changeof each parameter a(cid:11)ects strength. Finally, a strength data display shouldshow the comparative capabilities of di(cid:11)erent individuals or populations.2.8.2 Design of Strength Data DisplaysAn e(cid:11)ective strength data display conveys the meaning of the parameters andtheir values [Eva85, CBR, EC86, EPE88, MKK88]. To design displays that+can portray the multi-dimensional nature of strength e(cid:11)ectively, we use thehuman (cid:12)gure as a context [Wei90].Thestrength data display system evaluates the strength (cid:12)elds in theJackcurrent (cid:12)gure’s body de(cid:12)nition (cid:12)le. The strength data for the people whoare related (in the database sense) to the given (cid:12)gure displayed on the screencan be obtained by querying the strength database. For example, supposeFred is a NASA crewman. The required joint strength data for Fred canbe calculated from the equations stored in the strength (cid:12)elds of Fred’s bodyde(cid:12)nition (cid:12)le. However, any strength data for other people in the NASAcrewmen population must be queried from the strength database.If ende(cid:11)ector forces are not stored in the strength database, they can be calculated.Strength Box DisplaySix orthogonally intersecting rods are used to show the end e(cid:11)ector forces indi(cid:11)erent directions at a given body posture. They can also be used to showmuscle group strengths of a three DOFs joint at a given body con(cid:12)guration.The direction of the given force (or strength) is shown by the position andthe orientation of the rod. It is also indicated by the text at the side of rods.The length of each strength rod shows the magnitude of the force/strengthvalue at a given direction of movement. As the body con(cid:12)guration changes,the rod lengths are changed to re(cid:13)ect changing strength.The strength rods in Figure 2.7 show the hand forces atup, down, left,right, push,pullanddirections of the displayed human (cid:12)gure at the cur-62CHAPTER 2. BODY MODELINGrently displayed body con(cid:12)guration. The internal box represents average fe-male strength while the outer box represents average male strength given thesame posture. The length of the interior segment of the rod represents theaverage strength value of females and the length of the exterior segment of therod indicates the di(cid:11)erence between the female and male average strengths.Thus, the average male strength is represented by the length of the wholerod. The box itself is used to show the boundary of the maximum forces ofthe range of motions in six di(cid:11)erent directions. The length of the red lineattached to the end of each rod shows the di(cid:11)erence between the maximumforce and the force of the current body position shown on the screen at a givendirection. The resulting display of the strength rods and the box is called thestrength boxdisplay. The user can (cid:12)nd the body con(cid:12)guration associated withthe maximum force for the entire range of motions at a given direction by in-teractively adjusting the body posture until the end of the given rod touchesthe corresponding edge of the box.This strength box display can be modi(cid:12)ed in many ways to show otherstrength or force data. For example, it can be changed to display data for aone or two DOF joint or to show the e(cid:11)ects of di(cid:11)erent parameters:(cid:15)Two collinear rods can be used to display the strength data for a oneDOF joint, and four coplanar rods can be used to display the strengthdata of a two DOF joint.(cid:15)Each rod can be divided into one or more segments by using two di(cid:11)erentcolors to show the comparative muscle group strength of a strengthparameter that has one or more values, respectively. This is shown fora male/female comparison in Figure 2.7. Strengths corresponding todominant and nondominant hands can also be shown in this fashion.A three-segment rod can be used to show strengths of three di(cid:11)erentpercentiles of a population.(cid:15)A two-segment rod can be modi(cid:12)ed to show the maximum strength andrequired torque of a joint. For a given load applied at an end e(cid:11)ector,we can calculate the required torque via static force analysis at eachjoint in the active joint chain.(cid:15)A trace can be incorporated in the strength display to show the move-ment path of an end e(cid:11)ector. Two di(cid:11)erent trace colors can be usedto show the safe region where the maximum strength (of all joints) isgreater than the required torque (green), and the forbidden region wherethe maximum strength (of at least one joint) is less than the requiredtorque (red). (Figure 2.8). The number \20 lbs." written on the cubeat the end e(cid:11)ector is a reminder of the current load.The strength box display is mainly designed to show the strengths for a\single" joint or the forces of \one" end e(cid:11)ector. Although it can also beused to display strengths for more than one joint by showing them in two or2.8. STRENGTH AND TORQUE DISPLAY63Figure 2.7: Strength Box Display of Hand Forces for Males and Females of aPopulation.more strength box displays, it may not be easy to compare the values acrossdi(cid:11)erent boxes. To e(cid:11)ectively display strengths for more than one joint or ende(cid:11)ector, we use the strength bar display.Strength Bar DisplayThedisplay is used to show forces of end e(cid:11)ectors or strengthsstrength barof joints in a given body chain. Figure 2.8 illustrates a strength bar displaythat shows the maximum muscle group strengths and required torques ofjoints in the highlighted body chain when the hand is holding a load of 20lbs. If the maximum muscle group strength is greater than the correspondingrequired torque, the interior segment of a bar shows the required torque inred. Otherwise, the interior segment of a bar shows the maximum musclegroup strength in green. The exterior segment of a bar is always used toshow the di(cid:11)erence between the maximum strength and the required torque.If the maximum strength is greater than the required torque, the exteriorsegment is shown in green, otherwise it is shown in red. The bar (joint) withthe required torque exceeding the maximum strength can be indicated by64CHAPTER 2. BODY MODELINGFigure 2.8: Strength Bar Display for Maximum Strengths and RequiredTorques.highlighting. A one-segment bar in green indicates that the required torqueis zero; the length of the green bar shows the value of the maximum strength.Similarly, a one-segment bar in purple indicates that the required torque isequal to the maximum strength.The strength bar display can also be modi(cid:12)ed to show strength or forcedata in di(cid:11)erent applications. We list some simple extensions in the following.(cid:15)Similar to a rod in the strength box display, each bar in the display canalso be divided into a number of segments by using various colors toshow strengths corresponding to di(cid:11)erent values of a parameter.(cid:15)Multiple viewports can be used to display the strengths associated withdi(cid:11)erent values of a parameter. For example, we can use two viewportsto show the strength bar displays for the dominant and non-dominanthand of a given individual. Comparing strength values from di(cid:11)erentstrength bar displays is as easy as comparing strength values within astrength bar display because the 2D screen location of the display doesnot a(cid:11)ect the visual length of each bar.2.8. STRENGTH AND TORQUE DISPLAY65Figure 2.9: Torques Computed by Static Analysis.The strength bar display can be used to show strengths of any number ofjoints in a body chain. It gives a very clear view of the simultaneous e(cid:11)ects ofthe body con(cid:12)guration on several muscle group strengths. It also shows thesafe and forbidden regions of each joint in a body chain. This display methoddoes not depend on a particular strength model: it only shows whatever datais currently available for the (cid:12)gure.Whole body strength and torque display11Using static analysis, torques throughout the body may be computed givenany posture and any loading condition. The strength bar display can showthe individual joint torques (Figure 2.9). If suitable strength data for eachmuscle group is available, then whole body loading may be assessed.Of course, with so many joints the interpretation of multiple graphs be-comes di(cid:14)cult. To alleviate this problem we map the strength or torque datadirectly onto the contour body (Figure 2.10). The mapping interpolates12a given value at a joint onto the adjacent contour body polygon vertices.A parameter controls how far from the joint the mapped color will propa-gate. A typical color scale is based on mapping zero torque load to whiteand maximum strength to blue. Reacted torques exceeding the joint’s max-imum strength are colored red. Since only one attribute can be selected for1112Hyeongseok Ko, Susanna Wei, Michael KwonThis will also work for the simpler polyhedral body but is much less interesting.66CHAPTER 2. BODY MODELINGFigure 2.10: Contour Body with Color Coded Segments Representing TorqueLoad.a joint, the maximum load at multiple DOFs is used to select the color. Asthe mapping and color interpolation across each polygon take advantage ofthe Silicon Graphics workstation display speed, the visualization may even beinteractively observed as the posture and loads change.Chapter 3Spatial InteractionThis chapter describes the basic architecture of theinteractive system.JackThe primary tools available to theuser involve direct manipulation of theJackdisplayed ob jects and (cid:12)gures on the screen. With articulated (cid:12)gures, move-ment of one part will naturally a(cid:11)ect the position of other parts. Constraintsare used to specify these relationships, and an inverse kinematics algorithm isused to achieve constraint satisfaction. As a consequence of user actions, cer-tain global postural manipulations of the entire human (cid:12)gure are performedby the system. This chapter presents the direct spatial manipulations o(cid:11)eredinand shows how constraints are de(cid:12)ned and maintained. One partic-Jackular application of the body constraints is included: the generation of thereachable workspace of a chain of joints.3.1 Direct Manipulation3D direct manipulation is a technique for controlling positions and orienta-tions of geometric ob jects in a 3D environment in a non-numerical, visualway. It uses the visual structure as a handle on a geometric ob ject. Directmanipulation techniques derive their input from pointing devices and providea good correspondence between the movement of the physical device and theresulting movement of the ob ject that the device controls. This iskinestheticcorrespondence.Much research demonstrates the value of kinesthetically ap-propriate feedback [Bie87, BLP78, Sch83]. An example of this correspondencein a mouse-based translation operation is that if the user moves the mouse tothe left, the ob ject moves in such a way that its image on the screen movesto the left as well. The lack of kinesthetic feedback can make a manipula-tion system very di(cid:14)cult to use, akin to drawing while looking at your handthrough a set of inverting mirrors. Providing this correspondence in two di-mensions is fairly straightforward, but in three dimensions it is considerablymore complicated.The advantage of the direct manipulation paradigm is that it is intuitive:6768CHAPTER 3. SPATIAL INTERACTIONit should always be clear to the user how to move the input device to causethe ob ject to move in a desired direction. It focuses the user’s attention onthe ob ject, and gives the user the impression of manipulating the ob ject itself.3.1.1 TranslationSeveral techniques have been developed for describing three dimensional trans-formations with a two dimensional input device such as a mouse or tablet.Nielson and Olson [NO87] describe a technique for mapping the motion ofa two dimensional mouse cursor to three dimensional translations based onthe orientation of the pro jection of a world space coordinate triad onto thescreen. This technique uses a one-button mouse, and it compares the 2Ddisplacement of the mouse cursor to the screen pro jection of the six world co-ordinate axes and causes a di(cid:11)erential movement in world space along the axiswhose pro jection is closest to the mouse movement. For example, if the viewis positioned such that the world coordinateaxis points left, then movingxthe mouse to the left will cause a +translation. This provides good kines-xthetic correspondence, but it has problems if two of the axes pro ject onto thescreen close to one another, since it will not be able to distinguish betweenthe two. In other words, it is highly dependent on the view.3.1.2 RotationRotations are considerably more complex, but several techniques have beendeveloped with varying degrees of success. The most naive technique is to sim-ply use horizontal and vertical mouse movements to control the world spaceeuler angles that de(cid:12)ne the orientation of an ob ject. This technique pro-vides little kinesthetic feedback because there is no natural correspondencebetween the movements of the mouse and the rotation of the ob ject. A betterapproach, described by Chen et al [CMS88], is to make the rotation angles ei-ther parallel or perpendicular to the viewing direction. This makes the ob jectrotate relative to the graphics window, providing much greater kinestheticfeedback.The problem with screen-space transformations is that it is impossible tomake movements around either the global or local axes.In an integratedgeometric environment, it is more common to move ob jects relative to eitherthe global or local coordinate frame, rather than along axes aligned with thescreen. For example, the simple task of raising an ob ject vertically requirestranslating along the globalaxis. Unless the view in the graphics window isyperfectly horizontal, the vertical direction in screen coordinates is not exactlyvertical. As another example, the task of moving a hand forward may requiremoving along an axis aligned with the body, not the screen.Evans, Tanner, and Wein [ETW81] describe a rotation technique thatsuggests a turntable on which ob jects sit. Movements of the mouse in circlesaround the origin of the turntable cause the turntable, and thus the ob ject,3.1. DIRECT MANIPULATION69to rotate. There must also be a way of positioning the turntable underneaththe ob ject.Chen, Mountford, and Sellen [CMS88] also describe a technique originallydeveloped by Evans, Tanner, and Wein [ETW81] known commonly as thevirtual sphere.This technique simulates the e(cid:11)ect of a trackball centeredaround the ob ject’s origin. You \grab" the trackball with the mouse androtate it much as you would rotate a physical trackball with a single (cid:12)nger.The virtual sphere is an e(cid:14)cient technique for certain operations, as Chenet al verify experimentally. However, since the rotation is not con(cid:12)ned to aspeci(cid:12)c axis, it can be di(cid:14)cult to rotate around a particular axis. It is nearlyimpossible to make precise rotations around global coordinate axes.Chen et al describe an experimental comparison between several tech-niques for 3D rotation. The sub jects were asked to rotate a geometric ob ject,in the shape of a house, to align it with a similar ob ject in a random orien-tation. They were measured for both speed and accuracy. The techniquesevaluated included several angle-based rotations with and without kinestheticcorrespondence, and the virtual sphere. The studies generally showed thatthe virtual sphere was the best, out-performing the others in both precisionand speed.The virtual sphere is good at \tumbling" ob jects, when the path of theirrotation is not important. This may be the case for ob jects (cid:13)oating in space.However, in an integrated modeling environment, the technique has somelimitations because it does not allow constrained movement. Because of itsfree-form nature, it is very di(cid:14)cult to rotate an ob ject around a single axis ata time, global or local, which is often required. For example, to turn around ahuman being standing on the (cid:13)oor requires rotating only around the verticalaxis. With the virtual sphere, it is nearly impossible to rotate precisely aroundonly one axis at a time.An improvement over the virtual sphere is proposed by Shoemake [Sho92].His \ArcBall" approach uses the visual correspondence between arcs on ahemisphere and 3D quaternions to de(cid:12)ne simultaneously both a rotation angleand axis. Any 2D screen point input has a well-de(cid:12)ned rotational value. TheArcBall appears best for tumbling ob jects, but a constrained axis formulationis essentially similar to thelocal rotation operation described below.Jack3.1.3 Integrated SystemsBier’stechnique [Bie86, Bie87, Bie90] simulates gravity betweensnap-draggingob jects and takes advantage of surface geometry to constrain and controlob ject movements. The user (cid:12)rst positionsin space using a 3D cursorjackscalled the. The jacks are coordinate frames that serve as anchorsskitterfor other operations. The skitter slides along faces and edges, controlledby the mouse or through dials. The technique determines the position andorientation of the visible surface beneath the mouse in order to control theposition and orientation of the skitter. Jacks can be placed with the skitterand then used to specify rotation axes or end-points for angles.70CHAPTER 3. SPATIAL INTERACTION????Read mouse coordinates and button statusConvert to manipulation transformApply manipulation transform to the environmentRedraw graphics windowsFigure 3.1: The manipulation loop.This technique exploits the geometric structure of the ob jects, but it pro-vides little help for manipulating positions and orientations in the absence ofgeometry. This means that the technique does not work especially well formanipulating points in open space as is often required.3.1.4 The Jack Direct Manipulation OperatorThe 3D direct manipulation mechanism ininteractively describes a globalJackhomogeneous transform.Internally, this is called themanipulation trans-form.JackThere are many di(cid:11)erent commands inthat require the user toenter three-dimensional translational and rotational quantities. Each com-mand may interpret the transform in its own way, possibly mapping it to alocal coordinate system.The user manipulates this transform through the 3-button mouse, togetherwith theandkeys on the keyboard. The keys alter the in-SHIFTCONTROLterpretation of the mouse buttons. Each mouse button corresponds to anaxis in space, using a mapping scheme described below. The direct manipu-lation mechanism can alter the manipulation transform based on the selectedaxis by rotating around it, translating along it, or translating in a plane per-pendicular to the axis. This characterizes the three primitive types of directmanipulation: linear translation, planar translation, and rotation [PB88]. Themanipulation procedure is a loop, shown in Figure 3.1, that continues untilthe user terminates it.Theuser interface is modal: each manipulation command placesJackJackin a mode where the mouse buttons and keyboard keys are interpreted3.1. DIRECT MANIPULATION71SHIFTCONTROLmouse buttonsactionleftlinear transl along globalaxismiddlelinear transl along globalaxisrightlinear transl along globalaxisleft and middleplanar transl in globalplanexyleft and rightplanar transl in globalplanexzmiddle and rightplanar transl in globalplaneyzCONTROLleftrotation around globalaxisCONTROLmiddlerotation around globalaxisCONTROLrightrotation around globalaxisSHIFTleftlinear transl along localaxisSHIFTmiddlelinear transl along localaxisSHIFTrightlinear transl along localaxisSHIFTleft and middleplanar transl in localplanexySHIFTleft and rightplanar transl in localplanexzSHIFTmiddle and rightplanar transl in localplaneyzSHIFTCONTROLleftrotation around localaxisSHIFTCONTROLmiddlerotation around localaxisSHIFTCONTROLrightrotation around localaxisTable 3.1: Axis mappings for manipulation.xyyyyzxyyyyzas instructions to move the transform in question. How the movement isinterpreted depends upon the command. This mode is terminated by hittingthekey. While in the manipulation mode, the mouse buttons andESCAPEkeys behave as described below.The user interface for the manipulation operation encodes by default theleft, middle, and right mouse buttons to control translations along the,,xyandaxes, respectively, of the global coordinate frame. When the user presseszdown any mouse button, it enables translation along that axis. When theuser presses two mouse buttons, translation is enabled in the plane spannedby those two axes. With this technique, it is not possible to translate alongthree axes simultaneously, so pressing three buttons at once has no e(cid:11)ect.Rotation is signi(cid:12)ed by holding down thekey. In this case, theCONTROLmouse buttons are interpreted as rotations around the,, andaxes of thexyzglobal coordinate. Only one rotation button may be selected at once.The user can change the axes of translation and rotation to the localcoordinate frame of the manipulation transform by holding down theSHIFTkey. Thekey still signi(cid:12)es rotation, but the rotational axes areCONTROLlocal to the manipulation transform instead of the global axes. Table 3.1summarizes how the state of the keys and mouse buttons translates into thetransform axis.Jackrelies on a set of graphical icons to inform the user about the axes of72CHAPTER 3. SPATIAL INTERACTIONtranslation and rotation. The manipulation transform is drawn as a labeledsix-axis coordinate frame. The translation icon is a transparent arrow. Therotation icon is a spoked wheel. Each icon is overlaid against the ob jectsthemselves, but since they are transparent, they do not intrude too severely.The Mouse LineThe translation and rotation of the manipulation transform is determinedinteractively by the ray in the world coordinates that is cast through thelocation on the screen where the mouse cursor lies. This line in space isreferred to internally as the, and it can be easily computed by anmouse lineinversion of the viewing transform. The mouse line serves as ainto theprobeenvironment with which to move the manipulation transform. This notion ofa probe is useful in describing the implementation, although it is not one thatis visible to the user. The user has the feel of moving the ob ject itself.Linear and angular displacements are computed by intersecting the mouseline with lines and planes de(cid:12)ned by the origin of the manipulation transformand the translation or rotation axis using a scheme described below.Linear TranslationAs described above, linear translation takes place when the user presses onemouse button. The mouse may move anywhere on the screen, but the trans-lation is restricted to the particular axis, determined by which mouse buttonwas pressed. This axis pro jects to a line on the screen. The translation iconillustrates this line, and it also instructs the user to move the mouse in thatdirection on the screen. Ideally, the user moves the mouse exactly along thisline, but in practice the mouse will not follow the line precisely. The positionof the manipulation transform is the point along the translational axis that isnearest to the mouse line. Figure 3.2 shows the translation icon.Planar TranslationPlanar translation is actually somewhat simpler than linear translation be-cause its two DOFs more closely match those of the mouse. The plane oftranslation passes through the origin of the manipulation transform, and isspanned by the two axes de(cid:12)ned by the selected mouse buttons. The tech-nique is to intersect the mouse line with the plane of translation and supplythe point of intersection as the origin of the manipulation transform. Thismeans that the ob ject automatically goes to the location in the plane that liesunderneath the mouse cursor. Figure 3.3 shows the planar translation icon.The user can translate in the global or local,, orplanes, but inxyxzyzpractice the linear and planar translation techniques provide a comfortablepattern of use involving only planar translation in theplane. This isxzthe horizontal plane, and it is the most intuitive to visualize. The user cancomfortably translate ob jects in the horizontal plane using planar translation,and then raise and lower them using linear translation along theaxis.y3.1. DIRECT MANIPULATION73Figure 3.2: Linear translation.Figure 3.3: Planar translation.74CHAPTER 3. SPATIAL INTERACTIONFigure 3.4: Rotation.RotationThe user interface for rotation requires the user to move the mouse around incircles on its pad. This is similar in some ways to the turntable technique of[ETW81] described earlier, except that the turntable used a rotation angle inscreen coordinates, not world coordinates, making the angular displacementindependent of the viewing direction. Theoperator provides a moreJackdirect feel over the rotation because it gives the sense of actually holding onto the wheel.The three mouse buttons are encoded as rotation around the,, andxyzaxes. When the user presses down on a button, a wheel is displayed atthe origin of the manipulation transform describing the rotational axis. Thiswheel lies in the plane in which the rotation is to take place, with the originof the wheel at the rotational axis. Then a vector is drawn from the currentintersection point between the plane and the mouse line. This vector forms anextendedof the wheel, and as the user moves the mouse around in thisspokeplane,computes a new spoke and then measures the angular distanceJackbetween it and the original spoke. This determines the angle of rotation. Thesensation that the user gets is one of twisting a crank by moving the mousearound in circles on the screen. Figure 3.4 shows the rotation wheel icon.If the rotation involves ajoint with joint limits, the inadmissiblePeabodysector of the rotation wheel is colored red. Mouse movements will cause thewheel to rotate beyond its limits, but the rotated segment will not move pastthe limit. Thus the semantics of the user’s interaction are consistent with the3.2. MANIPULATION WITH CONSTRAINTS75free rotation case, yet ob ject behavior is natural.3.2 Manipulation with ConstraintsThe term \constraint" has many meanings and applications. Some researchersuse it to mean a very low level concept. Isaacs and Cohen [IC87] use the termto mean essentially any kinematic control over a joint or group of joints dur-ing dynamic simulation. The constraint removes DOFs from the system ina straightforward way. Witkin and Kass [WK88] and Girard [Gir91] use theterm to mean a global optimization criterion, such as minimum expendedenergy. Sometimes, the term means an desired relationship, which in com-puter graphics usually implies a geometric one. This is the case of Nelson’sJuno system [Nel85]. Many researchers use the term to mean speci(cid:12)cally adesiredrelationship, that is, goals for reference points. This is usuallyspatialthe meaning in physically based modeling, as in the dynamic constraints ofBarzel and Barr [BB88].Constraints that mean geometric goals may be interpreted with an ad-ditional temporal component. Most constraint-based systems like those justmentioned feature the ability to vary the e(cid:11)ect of constraints over time. Giventhis, it is rather nebulous whether the temporal component is a part of theconstraint de(cid:12)nition. We feel that it is better to view a constraint instanta-neously as a static entity, although its parameters can be changed over timeby some means external to the constraint itself.3.2.1 Postural Control using ConstraintsMost formulations of positioning algorithms in robotics and computer anima-tion are principally concerned with motion | smooth motion. In robotics,this is necessary because jerky motion could damage a manipulator. In an-imation, jerky motion looks bad. For postural control the motion is not asimportant because the static posture is the principal ob jective. This meansthat for interactive postural control, it may be possible to entertain optionswhich are not available to robotics and animation.In particular, the requirements for postural control are, (cid:12)rst, that thetechnique accommodate massively redundant (cid:12)gures, and second, that it per-form fast enough for interactive manipulation, even with complex, highly con-strained (cid:12)gures. The third concern is that it generate smooth movement. Theinteractive postural control process consists of placing the (cid:12)gure in a sequenceof postures closely spaced in time and space, giving the illusion of motion forpurposes of assisting in the postural speci(cid:12)cation process.We use inverse kinematics for posture determination [ZB89]. It dispenseswith physical interpretations and solves a minimization problem for pure func-tions using a nonlinear programming algorithm. The approach uses a variable-metric optimization technique. The function it minimizes is a de(cid:12)ned througha linear combination of kinematic constraints as de(cid:12)ned below. The ob jective76CHAPTER 3. SPATIAL INTERACTIONfunction describes positions, not velocities. This approach essentially treatsthe (cid:12)gure a purely geometric entity rather than a physical one. It does nottake into account overtly the (cid:12)gure’s mass and inertia, though we will see thatit can be used to control the center of mass. Our implementation of inversekinematics is highly e(cid:14)cient and is capable of controlling a complex (cid:12)guremodel with large numbers of constraints.The approach that we advocate for de(cid:12)ning postures is somewhat similarto the energy constraints of Witkin, Fleischer, and Barr [WFB87], althoughthere are some important distinctions. Their approach models connectionsbetween ob jects using energy functions. The energy functions do not measuremechanical energy, but are simply constructed to have zeros at the properlocations and have smooth gradients so that the ob ject can follow the gradienttowards the solution. This provides the animation of the solution process. Theuser interface of Witkin, Fleischer, and Barr’s system allows the user to specifythese energy functions and turn them on. This causes a sudden increasein energy and thus causes the model to begin descending in the directionof the gradient of the energy function. One drawback of this approach isthat the timestep of the iterative process must be su(cid:14)ciently small to ensureconvergence. This is particularly a problem in the case of articulated (cid:12)gures.Witkin, Fleischer, and Barr’s formulation of the gradient descent algorithmdoes not permit jointed mechanisms, so they must model joints as constraints.The joints must have a very steep energy function to ensure that they nevercome apart. This means that the timestep must be very small, making thesystem of equations very sti(cid:11).Another problem with Witkin, Fleischer, and Barr’s approach, from ahigher level point of view, is that the user’s notion of time is embedded in thealgorithm. The energy function de(cid:12)nes the path of the end e(cid:11)ectors towardstheir ultimate destinations because the gradient of the energy function de-termines the direction of movement. We remove the notion of time from thenumerical process and put it in a higher level control scheme. This controlscheme decides on discrete locations for the goals for end e(cid:11)ector referencepoints, closely spaced in space and time | the user’s notion of time. The in-verse kinematics algorithm solves an instantaneous positioning problem. Thecontrol scheme has two components, the interactive manipulation previouslydiscussed and the behaviors described in Chapter 4.The inverse kinematics procedure itself has no notion of time.It usesthe joint angles of the(cid:12)gures as the variables of optimization.PeabodyThis means that the search step can be signi(cid:12)cantly larger, and thus theconvergence is faster. The control scheme guides the movement in cartesianspace, rather than in the space of the energy function. It is still convenient tointerpret the positioning criteria as a potential energy function, but the controlscheme ensures that the (cid:12)gure is always very near a state of equilibrium.Using the example of a human arm reaching for a switch, the techniqueof Witkin, Fleischer, and Barr would model the reach by de(cid:12)ning an energyfunction with a zero when the hand is on the switch, that is, a constraintfor the hand to lie at the switch. If the arm begins by the side of the body,3.2. MANIPULATION WITH CONSTRAINTS77the arm would see a sudden increase in energy when the constraint becomesactive and would immediately begin to move in a way to minimize the energy.This would cause the hand to begin to move towards the switch. The velocitywould be controlled through the intensity of the potential energy function.Using inverse kinematics, the higher level control scheme computes suc-cessive locations for the hand, starting at its current location and progressingtowards the switch. The number of steps and the spacing between them isnot the domain of the inverse kinematics algorithm but of the control scheme.A kinematic constraint describes the position of the hand and the algorithm’sparameters at each time step. The inverse kinematics algorithm is invoked ateach time step to place the hand at the desired location.3.2.2 Constraints for Inverse KinematicsThenotion of a kinematicde(cid:12)nes acoordinate frame, anJackconstraintgoalend e(cid:11)ectorset of jointscoordinate frame, awhich control the end e(cid:11)ector,and anwhich measures the distance between the goal andobjective functionthe end e(cid:11)ector. The set of joints is usually de(cid:12)ned in terms of astarting joint,and the joint set then consists of the chain of joints between the starting jointand the end e(cid:11)ector. We can think of individual constraints as de(cid:12)ning a po-tential energy, measured by the ob jective function, much like that of Witkin,Fleischer, and Barr. The constraint is satis(cid:12)ed when the ob jective functionis minimized, although the function need not be zero. The potential energyof several constraints is a weighted sum of the energies from the individualconstraints.The ob jective function of a constraint has separate position and orientationcomponents. The potential energy of the constraint is a weighted combinationof the two, according to the constraint’s. Theposition/orientation weightweight may specify one or the other, or a blending of the two. Our inversekinematics procedure provides the following ob jective types [ZB89]:pointThe point-to-point distance between the end e(cid:11)ector andthe goal.lineThe point-to-line distance between the end e(cid:11)ector and a linede(cid:12)ned in the goal coordinate frame.planeThe point-to-plane distance between the end e(cid:11)ector anda plane de(cid:12)ned in the goal coordinate frame.frameThe orientational di(cid:11)erence between the end e(cid:11)ector frameand the goal frame. This measures all three orientationaldirectionThe orientational di(cid:11)erence between a vector de(cid:12)ned inthe end e(cid:11)ector frame and a vector de(cid:12)ned in the coordinateDOFs.frame of the goal.aimA combined position/orientation function designed to \aim"a reference vector in the coordinate frame of the end e(cid:11)ector78CHAPTER 3. SPATIAL INTERACTIONtowards the position of the goal. This is used mostly foreye and camera positioning, but it could also be used, forexample, to point a gun. This should never be done in thepresence of multiple human (cid:12)gures, of course, particularlychildren.3.2.3 Features of ConstraintsEach constraint inhas its own set of joint variables, so the control ofJackeach constraint can be localized to particular parts of the (cid:12)gure. Since thejoint angles are the variables of optimization, this means that the algorithmoperates on chains of joints, ultimately terminated at the root of the (cid:12)gurehierarchy. This implies that the root of the (cid:12)gure remains (cid:12)xed during thepositioning process. This makes the placement of the (cid:12)gure root particularlyimportant. One of the ma jor components ofis that the actualPeabodyde(cid:12)nition of a (cid:12)gure does not include the (cid:12)gure root. Instead, the root is adynamic property which can be changed from time to time. Since the inversekinematics algorithm operates on chains emanating from the root, the inversekinematics algorithm cannot change the location of the root.Our inverse kinematics algorithm works very well provided that it doesn’thave to search too far for a solution, although it will converge from any startingpoint. The farther it has to search, the more likely it is to produce largechanges in the joint angles.In geometric terms, this means that the goalsshould never be too far away from their end e(cid:11)ectors, lest the interior segmentsof the joint chains move too far. This also relieves the problem of gettingtrapped in a local minimum because hopefully the higher level control strategywhich is specifying the goal positions will do so in a way to lead the (cid:12)gureaway from such conditions.The elegance of the potential energy approach, like that of Witkin, Fleis-cher, and Barr, is that the constraints can be overlapped. This means thatit is acceptable to over-constrain the (cid:12)gure. The posture which the algo-rithm achieves in this case yields the minimum energy state according to theweighting factors between the constraints. This provides a way of resolvingthe redundancies in a massively redundant (cid:12)gure: use lots of constraints,and don’t worry too much about whether the constraints specify con(cid:13)ictinginformation.3.2.4 Inverse Kinematics and the Center of MassThe center of mass of an ob ject is one of its most important landmarks becauseit de(cid:12)nes the focal point for forces and torques acting on it. The center ofmass of an articulated (cid:12)gure such as a human (cid:12)gure is particularly signi(cid:12)cantbecause its location relative to the feet de(cid:12)nes the state of balance. Thisis critical for human (cid:12)gures, because so many aspects of the movement of ahuman (cid:12)gure are dictated by the need to maintain balance. The center ofmass of is, of course, a dynamic property, but it is possible to manipulate it3.2. MANIPULATION WITH CONSTRAINTS79in a purely kinematic way and thus produce some of the e(cid:11)ects of dynamicsimulation without the extra cost.Methods of Maintaining BalanceOne approach to maintaining balance of an articulated (cid:12)gure is to root the(cid:12)gure through its center of mass. The center of mass is a dynamic featureof a (cid:12)gure, so rooting the (cid:12)gure through the center of mass means that eachtime the (cid:12)gure moves, the center of mass must be recomputed and the (cid:12)g-ure’s location updated so that the center of mass remains at the same globallocation.This approach works, but it does not give good control over the elevationof the center of mass, since the center of mass is e(cid:11)ectively constrained toa constant elevation as well as location in the horizontal plane. The (cid:12)gureappears to dangle as if suspended from its waist with its feet reaching out forthe (cid:13)oor. This is particularly true during an operation in which the centerof mass normally goes down, such as bending over. In order for the balancebehavior to function naturally, the elevation of the center of mass must beallowed to (cid:13)oat up and down as required by the movement of the feet. Thisis more appropriately handled through a constraint.Kinematic Constraints on the Center of MassBalancing a (cid:12)gure is achieved by constraining the center of mass to remaindirectly above a point in the support polygon. The constraint designates asingle point as the balance point rather than using the entire support polygonbecause this gives control over the placement of the point within the polygon.This allows the (cid:12)gure’s weight to shift side to side or forward and backward,without moving the feet.Jackassociates the center of mass logically with the lower torso region ofthe human (cid:12)gure, and it uses this as the end e(cid:11)ector of the constraint, withthe ankle, knee, and hip joints of the dominant leg as the constraint vari-ables. During the constraint satisfaction process at each interactive iteration,the center of mass is not recomputed. Since the center of mass belongs logi-cally to the lower torso, its position relative to the torso remains (cid:12)xed as theinverse kinematics algorithm positions the ankle, knee, and hip so that thepreviously computed center of mass point lies above the balance point. Thereare generally other constraints active at the same time, along with other pos-tural adjustments, so that several parts of the (cid:12)gure assume di(cid:11)erent posturesduring the process.After the constraints are solved,recomputes the center of mass. ItJackwill generally lie in a di(cid:11)erent location because of the postural adjustments,indicating that the (cid:12)gure is not balanced as it should be. Therefore, the con-straints must be solved again, and the the process repeated until the balancecondition is satis(cid:12)ed. In this case the structure of the human (cid:12)gure helps.Most of the postural adjustments take place on the (cid:12)rst iteration, so on sub-80CHAPTER 3. SPATIAL INTERACTIONsequent iterations the changes in the center of mass relative to the rest of thebody are quite minor.measures the distance that the center of massJackchanges from one iteration to the next, and it accepts the posture when thechange is below a certain threshold. Although it is di(cid:14)cult to guarantee theconvergence theoretically, in practice it seldom takes more than two iterationsto achieve balance.3.2.5 Interactive MethodologyThere are several possibilities for overcoming the problems with redundanciesand local minima. One is to incorporate more information into the ob jectivefunction, modeling such factors as strength, comfort, and agent preference(Section 5.3). This is an important addition, although it adds signi(cid:12)cantlyto the computational complexity of the constraint solving procedure.’sJacktechnique is to provide the positional input to the inverse kinematics algo-rithm with the 3D direct manipulation system.allows the user to in-Jackteractively \drag" goal positions and have the end e(cid:11)ector follow[PZB90]. Inthis case, the geometric information obtained by the mouse at each iterationof the manipulation process is applied to the goal position of a constraint,and the inverse kinematics algorithm is called to solve the constraints beforethe graphics windows are redrawn. Alternatively, the user can move a (cid:12)g-ure which has end e(cid:11)ectors constrained to other ob jects. Each case causes arelative displacement between the end e(cid:11)ector and the goal.Interactive DraggingThis dragging mechanism is a modi(cid:12)ed version of the basic direct manipula-tion scheme described in Section 3.1. After selecting the parameters of theconstraint, the manipulation procedure works as shown in Figure 3.5. The in-verse kinematics procedure is invoked at every iteration during the interactivemanipulation.This is a very e(cid:11)ective and e(cid:14)cient tool for manipulation for several rea-sons. Because of the incremental nature of the interactive manipulation pro-cess, the goals never move very far from one iteration to the next, as necessary.The algorithm still su(cid:11)ers from problems of local minima, but since the usercan drag the end e(cid:11)ector around in space in a well-de(cid:12)ned and easy to controlway, it is relatively easy to overcome these problems by stretching the (cid:12)gureinto temporary intermediate con(cid:12)gurations to get one part of the (cid:12)gure posi-tioned correctly, and then dragging the end e(cid:11)ector itself into the (cid:12)nal desiredposition. This takes advantage of the user’s abilities, because local minimacan be easy for the user to see but di(cid:14)cult for the system to detect and avoid.A common example of this dragging technique involves the elbow. Theuser may initially position the hand at the proper place in space but then (cid:12)ndthat the elbow is too high. If this is the case, the user can extend the handoutwards to drag the elbow into the correct general region and then drag thehand back to the proper location.3.2. MANIPULATION WITH CONSTRAINTS81Read mouse coordinates and button status?????Convert to manipulation transformApply manipulation transform to goalInvoke inverse kinematics algorithmRedraw graphics windowsFigure 3.5: Interactive Dragging.Interactive TwistingAnother e(cid:11)ective feature of the direct manipulation interface is the use oforientation constraints, particularly the weighted combination of position andorientation. In this case, the orientation of the goal is signi(cid:12)cant as well asthe position, so the user may manipulate segments in the interior of the jointchain by twisting the orientation of the goal and thus the end e(cid:11)ector. Thisis especially helpful because of the di(cid:14)culty the user encounters in visualizingand numerically describing rotations which will achieve a desired orientation.The above example of the elbow position may be handled this way, too. Bytwisting the desired orientation of the hand, the interior of the arm can berotated up and down while the hand remains in the same location. Thisachieves in real-time a generalization of the \elbow circle" positioning schemeimplemented by Korein [Kor85].This raises an interface issue concerning the relationship between the ac-tual orientation of the end e(cid:11)ector coordinate frame and the manipulationtransform. The manipulation technique described in Section 3.1 allows theuser to translate and rotate a 6D quantity which now guides the positionand orientation of the end e(cid:11)ector. We noted, however, that this techniqueis not so good at choreographing smooth movements through space. Themovement tra jectory generated by the technique consists of intermittent seg-82CHAPTER 3. SPATIAL INTERACTIONFigure 3.6: The Orientation Constraint Icon.ments of straight-line translations and rotations. As the user translates themanipulation transform, its orientation remains (cid:12)xed, and vice versa. Is thisappropriate behavior for the end e(cid:11)ector as well?If the end e(cid:11)ector is sensitive to orientation, then translating the manip-ulation transform means that the end e(cid:11)ector will translatebut wil l try keepthe same global orientation.Typically, the user can quickly arrive at a goalposition for the end e(cid:11)ector which is not achievable.Positional di(cid:11)erences are easy to visualize; orientational di(cid:11)erences arenot. It is easy to manipulate a positional goal which is outside of the reach-able space of the end e(cid:11)ector. We can intuitively think of a spring or rubberband pulling the end e(cid:11)ector towards the goal. Orientational di(cid:11)erences aremuch harder to visualize. Even though it may be easy to conceptualize \ro-tational springs," in practice is it very di(cid:14)cult to apply that intuition to thegeometric model. If the goal is very far outside of the reachable space of theend e(cid:11)ector along, all correspondence between the ori-angular dimensionsentation of the goal and the end e(cid:11)ector gets quickly lost.illustratesJackorientational di(cid:11)erences through rotationshown in Figure 3.6, whichfans,are icons to illustrate how an ob ject must rotate to achieve a desired orienta-tion, but no amount of graphical feedback can help when the di(cid:11)erences arelarge. Therefore, it is absolutely essential that the orientation of the goal {the manipulation transform | not deviate too far from the end e(cid:11)ector.Jacko(cid:11)setsolves this problem through an orientationto the goal whichcan be adjusted during the manipulation process. This o(cid:11)set is a relativetransform which is applied to the manipulation transform to rotate it into the3.3.INVERSE KINEMATIC POSITIONING83true orientation of the goal as supplied to the inverse kinematics algorithm.The end e(cid:11)ector dragging mechanism resets this o(cid:11)set each time a translationor rotation is completed during the manipulation process, that is, each time amouse button comes up and the movement stops. This means that each timethe user begins to rotate or translate the goal, the manipulation transformstarts out from the current orientation of the end e(cid:11)ector. This prevents theuser from getting lost in the orientation di(cid:11)erence.This simulates the e(cid:11)ect of a spring-loaded crank which applies a torque toan ob ject, but only as long as the user holds down the mouse button. Whenthe mouse button comes up, the torque disappears so that it doesn’t continueto have a undesirable e(cid:11)ect. This lets the user control the goal through aratcheting technique of applying short bursts of rotation.Manipulation with ConstraintsThe nature of the 3D direct manipulation mechanism allows the user to inter-actively manipulate only a single element at a time, although most positioningtasks involve several parts of the (cid:12)gure, such as both feet, both hands, etc.In addition to interactively dragging a single end e(cid:11)ector, there may be anynumber of other kinematic constraints. These constraints are persistent rela-tionships to be enforced as the (cid:12)gure is manipulated using any of the othermanipulation tools. By (cid:12)rst de(cid:12)ning multiple constraints and then manipu-lating the (cid:12)gure, either directly or with the dragging mechanism, it is possibleto control the (cid:12)gure’s posture in a complex way.This mechanism involves another slight modi(cid:12)cation to the direct manip-ulation loop, shown in Figure 3.7. Step #4 may cause the end e(cid:11)ectors tomove away from their goal positions. The inverse kinematics algorithm instep #5 repositions the joints so the goals are satis(cid:12)ed.3.3Inverse Kinematic Positioning1Having modeled the articulated (cid:12)gure with segments and joints we need todeal with the issue of how to manipulate it. There are two basic problems:given all joint angles, how to compute the spatial con(cid:12)guration and, con-versely, given a certain posture, what values should be assigned to joint angles.The (cid:12)rst problem, forward kinematics, is simply a matter of straightforwardtransformation matrix multiplication [FvDFH90]. The second, inverse kine-matic problem, is much harder to solve.Inverse kinematics is extremely important in computer animation, sincethe spatial appearance, rather than the joint angles, interest an animator. Agood interactive manipulation environment such asmay simply hide theJacknumerical values of the joint angles from the user. In such an environment,where the user is concerned only with spatial con(cid:12)guration, the joint anglescan become merely an internal representation. The transformation from the1Jianmin Zhao.84CHAPTER 3. SPATIAL INTERACTION??????Read mouse coordinates and button statusConvert to manipulation transformApply manipulation transform to environmentRecompute global end e(cid:11)ector and goal transformsInvoke inverse kinematics algorithmRedraw graphics windowsFigure 3.7: Manipulation with Constraints.spatial con(cid:12)guration into the joint angles is carried out by the inverse kine-matic computation so an e(cid:14)cient implementation is most desirable.Inverse kinematics for determining mechanism motion is a common tech-nique in mechanical engineering, particularly in robot research [Pau81]. Inrobots, however, redundant DOFs are usually avoided. Moreover, the compu-tation is usually carried out on particular linkage systems [KH83, KTV90].+In contrast, an interesting ob ject in the computer animation domain { thehuman body { has many redundant DOFs when viewed as a kinematic mech-anism. Therefore a means for specifying and solving underconstrained posi-tioning of tree-like articulated (cid:12)gures is needed.We (cid:12)rst studied methods for kinematic chain positioning, especially inthe context of joint limits and redundant DOFs [KB82, Kor85]. Later wetried position constraints to specify spatial con(cid:12)gurations of articulated (cid:12)gures[BMW87]. A simple recursive solver computed joint angles of articulated(cid:12)gures satisfying multiple point-to-point position constraints.About the same time, Girard and Maciejewski used the pseudo-inverse of3.3.INVERSE KINEMATIC POSITIONING85the Jacobian matrix to solve spatial constraints [GM85]. The main formula is(cid:1)=(cid:18)J(cid:1)r+where (cid:1)is the increment of the joint angle vector, (cid:1)is the increment of the(cid:18)rspatial vector andis the pseudo-inverse of the Jacobian. For a largeJ@ r=@ (cid:18)+step size the method is actually the well-known Newton-Raphson method,which is not globally convergent and often needs some special handling (e.g.,hybrid methods [Pow70]). On the other hand, for a su(cid:14)ciently small stepsize, as Girard and Maciejewski suggested, excessive iterations are required.The inverse operation is normally very expensive; moreover, they did not dealwith joint limits.Witkinused energy constraints for positioning [WFB87]. The energyet alfunction they adopted is the sum of all constraints including ones for positionand orientation. Constraints are satis(cid:12)ed if and only if the energy functionis zero. Their method of solving the constraint is to integrate the di(cid:11)erentialequation:d(cid:18)t=dtE(cid:18)()= (cid:0)r()whereis the parameter (e.g., joint angle) vector,is the energy function(cid:18)Eof, and r is the gradient operator. Clearly, if() is the integral with some(cid:18)(cid:18)tinitial condition,(()) monotonically decreases with time. This integralE(cid:18)ttnot only gives a (cid:12)nal con(cid:12)guration which satis(cid:12)es the constraint in terms of(cid:18), but also a possible motion from the initial con(cid:12)guration which is driven bythe conservative force derived from the gradient of the energy function.Instead of associating energy functions with constraints, Barzel and Barrintroduced deviation functions such that a constraint is met if and only ifthe deviation function vanishes (= 0) [BB88]. They have presented variousconstraints, such as point-to-point, point-to-nail, etc., and their associateddeviation functions. A set of dynamic di(cid:11)erential equations are constructedto control the way by which the deviations vanish (e.g., exponentially in acertain amount of time). Constraint forces are solved which, along with otherexternal forces, drive geometric models to achieve constraints. They dealt withrigid bodies which may be related by various constraints. Witkin and Welch[WW90] used a dynamic method on nonrigid bodies. In all these approachesarticulated (cid:12)gure joints would be considered as point-to-point constraints,which are added to the system as algebraic equations. It is not unusual tohave several dozen joints in an highly articulated (cid:12)gure, which would add tothe number of constraint equations appreciably. Besides, a joint is meant tobe an absolute constraint. In another words, it should not compete with otherconstraints which relate a point in a segment of the (cid:12)gure with a point in thespace. This competition often gives rise to numerical instability. So thesemethods seem inappropriate to the highly articulated (cid:12)gure.Although the energy or dynamics methods may be used to solve the spa-tial constraint problem, they are also concerned with computing plausible(dynamically-valid) paths. For articulated (cid:12)gure positioning, the additionalwork involved in path determination is not so critical. So, we (cid:12)rst focus on86CHAPTER 3. SPATIAL INTERACTIONa more elementary version of the problem | to position the articulated (cid:12)g-ure into desired pose | hoping that the solution can be found much fasterand more robustly, and joint angles limits can be dealt with e(cid:11)ectively. Wewill see later in Chapter 5 how more natural motions may be generated byconsidering human strength models and collision avoidance.3.3.1 Constraints as a Nonlinear Programming ProblemA spatial constraint involves two parts. The part on the (cid:12)gure is called theend-e(cid:11)ector and its counterpart in the space is called the goal. A constraintis a demand that the end-e(cid:11)ector be placed at the goal. Since a goal alwaysimplies a related end-e(cid:11)ector, we may sometimes take goals as synonymousfor constraints.As is done with energy constraints, we create a function with each goalsuch that its value, when applied to the end e(cid:11)ector, represents the \distance"of the end-e(cid:11)ector from the goal. This distance need not be only Euclideandistance. Let us call this function \potential," because it is a scalar functionof spatial position or vectors. The vector (cid:12)eld generated by the negation of thegradient of the function could be interpreted as a force (cid:12)eld toward the goal.Depending on the goal types, an end-e(cid:11)ector on a segment can be a point, avector, a set of two vectors, or a combination of them, in the local coordinatesystem of the segment. Letdenote the potential function associated withPa goal andthe \location" of the end-e(cid:11)ector. The constraint is satis(cid:12)ed ifeand only if() vanishes. Clearly the end-e(cid:11)ector is a function of the vectorPeof all joint angles.A single constraint may not be su(cid:14)cient to specify a pose. For example,in addition to the position of the hand, the placement of the elbow is oftendesired.In positioning two arms, we may need to keep the torso in someorientation. The function associated with these conjunctively combined goalsis de(cid:12)ned as a weighted sum:G(cid:18)() =wG(cid:18)()ii(31):miX=1whereis the number of goals combined, andmG(cid:18)Pe(cid:18)i() =i(i())(32):where the subscriptrefers to theth goal, ands are weights on respectiveiiwiconstraints.Sometimes, goals may need be combined disjunctively. In other words, thecombined goal is achieved if and only if either constituent goal is achieved.For example, the interior of a convex polyhedron is the conjunction of innerhalf-spaces de(cid:12)ned by its surrounding faces, while the exterior is the disjunc-tion of those opposite half-spaces. Obstacle avoidance is a situation whereconstraining an end-e(cid:11)ector to the outside of a volume is useful. The function3.3.INVERSE KINEMATIC POSITIONING87associated with the disjunctively combined goal is de(cid:12)ned asG(cid:18)() = minG(cid:18)()i(33):i2f1;:::mgBy de(cid:12)nition of the function, the overall constraint is met if and only ifGGvanishes. Due to the physical constraint of the (cid:12)gure and the redundancyof DOFs, there may not besuch that() = 0, or there may be many such(cid:18)G(cid:18)(cid:18)(cid:18)’s. Therefore we attempt to (cid:12)nd a, sub ject to constraints on joint angles,which minimizes the function. Although constraints on joint angles may beGcomplicated, linear constraints su(cid:14)ce in most situations. Typical constraintsare represented as a lower limit and an upper limit for each joint angle. There-fore we formulate our problem as a nonlinear programming problem sub jectto linear constraints, i.e.,8== 12is:t: a(cid:18)b; i;; : : : ; l(34):a(cid:18)b; il; l; : : : ; k(cid:20)=+ 1+ 2imin()G(cid:18)TiTi<:where= 12are column vectors whose dimension is the total num-a; i;; : : : ; kiber of DOFs. The equality constraints allow for linear relations among thejoint angles. The lower limitand upper limiton, theth joint angle,lu(cid:18)iiiicontribute to the set of inequality constraints on:(cid:18)ii(cid:0)(cid:18)(cid:20) (cid:0)lii(cid:18)(cid:20).u3.3.2 Solving the Nonlinear Programming ProblemThere are many algorithms to solve problem (3.4). One e(cid:14)cient algorithm isbased on Davidon’s variable metric method with BFGS (Broyden, Fletcher,Goldfarb, Shanno) rank-two update formula [Fle70, Gol70, Sha70]. Rosen’spro jection method is used to handle the linear constraints on variables [Ros60,Gol69]. No matter what algorithm we choose, e(cid:14)ciency requires computationof the gradient as well as the value of the ob jective function whenis given.(cid:18)So (cid:12)rst we develop an e(cid:14)cient way to compute the gradient of, as well as thevalue of, the ob jective function() for (cid:12)gures with tree-like data structureG(cid:18)and various sorts of goals.Single ConstraintsSince the human (cid:12)gure has a tree-like structure, an end-e(cid:11)ector of a constraintdepends only on those joints which lie along the path from the root of the(cid:12)gure tree to the distal segment containing the end-e(cid:11)ector [BMW87]. Callthis path the. The constraint chain is illustrated in Figure 3.8.constraint chainFor simplicity, we assume each joint in Figure 3.8 has only one DOF. A jointwith multiple DOFs can be decomposed conceptually into several joints witheach having one DOF, i.e., zero distance from one joint to another is allowed.88CHAPTER 3. SPATIAL INTERACTION(cid:18)2(cid:18)i(cid:0)1hqHvhqhq(cid:28)A(cid:28)@(cid:28)(cid:28)(cid:10)A@uhqr(cid:1) (cid:1) (cid:1)(cid:10)H(cid:10)HH(cid:18)nH(cid:1)(cid:21)(cid:1)(cid:1)qend e(cid:11)ector segment(cid:28)(cid:28)(cid:10)A@(cid:2)(cid:14)(cid:28)(cid:28)(cid:2)(cid:10)A@(cid:28)A(cid:28)@(cid:2)
(cid:10)hqhqhqir(cid:18)1(cid:18)i(cid:1) (cid:1) (cid:1)Figure 3.8: Constraint Chain.The length of the constraint chain is the total number of joints along thechain; in Figure 3.8, it isn.This module is to compute(), as de(cid:12)ned in equation (3.2) and itsG(cid:18)igradient for a single constraint chain and its corresponding goal.In thissection, we only consider a single chain, so the subscriptis dropped.iG(cid:18)Pe(cid:18)() =(())(35):Notice thatandg(cid:18)() = r(cid:18)G@eT@ (cid:18)= ()r(3.6)Pewhereis the Jacobian matrix of the vector function(), i.e.,e(cid:18)@e@ (cid:18)@e@ (cid:18)@@ee@e= ((cid:1) (cid:1) (cid:1)) .12n@ (cid:18)@ (cid:18)@ (cid:18)These expressions suggest that end-e(cid:11)ectors and goals can be abstracted asfunctions((cid:1)) and((cid:1)) which, by pair, constitute constraints. The range ofePthe functiondepends on the type of the constraint, but it must be equaleto the domain of its respectivefunction, which measures the distance ofPthe current end-e(cid:11)ector to the goal. For the sake of numerical e(cid:14)ciency, werequire that those functions be di(cid:11)erentiable.By virtue of equations (3.5) and (3.6), two sub-modules, end-e(cid:11)ector andgoal modules, can be built separately. Note that the goal potentialdoesPnot depend on the (cid:12)gure model. Only the end-e(cid:11)ector functiondoes.eEnd-e(cid:11)ectorsThis module is the only one which depends on the model of the articulated(cid:12)gure. It turns out that it is easy to computeand, under our assumptione@e@ (cid:18)of rigidity of segments, for many useful types of constraint.3.3.INVERSE KINEMATIC POSITIONING89We only consider constraints where end-e(cid:11)ectors are points (for positions)or vectors (for orientations) on distal segments. Soconsists of either aeposition vector, one or two unit vectors attached on the distal segment’s,rvor a combination of them (see Figure 3.8). Because all the joints of the humanbody are revolute joints, we discuss here only revolute joints. (Translationaljoints are simpler and can be treated similarly.)Let theth joint angle along the chain be, the axis of this joint bei(cid:18)ui(a unit vector), and the position vector of theth joint be. The vectorsirriandcan be easily computed with cascaded multiplication of transformationvmatrices. The derivative can be computed as follows [Whi72]:@ ri@ (cid:18)@vi@ (cid:18)=urr(cid:2) ((cid:0)i)(3.7)=uv(cid:2).(3.8)These formulas are enough for the types of goals we have considered. Theparticular forms ofwill be explained with particular goals, since they mustematch the arguments of the goal potential((cid:1)).PGoal Typesconstraints).We have implemented several useful, as well as simple, types of goals (orPosition Goal.pThe goal is a pointin 3D space. The end-e(cid:11)ector is, corre-spondingly, a pointwhich sits on the distal segment of the constraintrchain, but it is not necessarily a leaf segment in the (cid:12)gure tree (seeFigure 3.8). The potential function is:Prpr() = ((cid:0))(39):2whereis the parameter of the function, and the gradient is:pr() = 2((cid:0)) .Prrp(310):rOrientation Goal.The orientation in the space is determined by two or-thonormal vectors. So the goal is de(cid:12)ned by a pair of orthonormalvectors, say,Accordingly, the end e(cid:11)ector is a pair of orthonormal vectorsfg .x; yggfeegx; yattached on the distal segment of the constraint chain.The potential function could be:Px; yxxyy() = ((cid:0))+ ((cid:0)eegege).2290CHAPTER 3. SPATIAL INTERACTIONIn combination with a positional goal, this function implies that onelength unit is as important as about one radian in angle. To enforce onelength unit compatible withdegrees in angle, we need to multiply thedpreviousbysuch thatPcd12(cid:25)=dcd360i. e. ,cd= 360(2) .=(cid:25)d(311):To be more general, our potential function is thenPx; yc() =ee(xgxe(cid:0))+c(yg(cid:0)ye).(312):dxdy2222The gradient isr() = 2e Px; yeec(xe(cid:0)g)x(3.13)xdx22r() = 2(cid:0)) .(3.14)e Px; yeec(yeygydyAny direction, such as, could be suppressed by settingto 0. Thisycdyis useful, for example, to constrain a person holding a cup of water whileattaining other constraints.Position/Orientation Goals.Position and orientation goals can be treatedseparately, but sometimes it is convenient to combine them together asa single goal. The potential function for position/orientation goal is justthe weighted sum of respectively goals:Pr; x; ywprwc() =((cid:0))+eepo(xgxe(cid:0))+wco(yg(cid:0)ye)(315):dxdy22222whereandare weights put on position and orientation respectivelywpwosuch thatwwpo+= 1 .The gradients rrand rare obvious from above.P;e Pe PrxyAiming-at Goals.pThe goal is a pointin the space, but the end-e(cid:11)ector isa vectorattached to the distal segment at(see Figure 3.8). The goalvris attained if the vectorpoints toward the point. This is useful, forvpexample, when we want to make the head face toward a certain point.The potential function is:Pr; vc() =((cid:0))v(316):22pr(cid:0)dk(cid:0)kprwhereis de(cid:12)ned in (3.11) and k (cid:1) k denotes the norm operation. Thecdgradient is:r() = 2(k(cid:0)k(cid:0) ((cid:0)) (cid:1)((cid:0)))k(cid:0)k(3.17)Pr; vcprvprvpr=prrd223r() = (cid:0)2(Pr; vc(cid:0)) .v(3.18)vd2pr(cid:0)k(cid:0)kpr3.3.INVERSE KINEMATIC POSITIONING91Line Goals.rThe goal is a line and the end-e(cid:11)ector is a point. The goal isto force the point to go along the line. Let the line be de(cid:12)ned by pointp(cid:23)and unit vectorsuch that the parametric equation of the line isThe potential function is:pt(cid:23)+.Prprpr(cid:23) (cid:23)() = (((cid:0)) (cid:0) ((cid:0)) (cid:1))(319):2and the gradient is:r() = 2((cid:1) ((cid:0))(cid:0) ((cid:0))) .Pr(cid:23)pr(cid:23)pr(320):rPlane Goals.rThe goal is a plane and the end-e(cid:11)ector is a point. Thepoint must be forced to lie on the plane. Let a point on the plane bepand the normal of the plane be. The potential function is:(cid:23)Prpr(cid:23)() = (((cid:0)) (cid:1))(321):2and the gradient is:r() = (cid:0)2(cid:1) ((cid:0)).Pr(cid:23)pr(cid:23)(322):rHalf-space Goals.The goal is one side of a plane and the end-e(cid:11)ector is apoint. The point is constrained to lie to one side of the plane. Let arpoint on the plane beand the normal of the plane bewhich pointsp(cid:23)to the half space the goal de(cid:12)nes. The potential function is:0if ((cid:0)) (cid:1)0pr(cid:23) <Pr() =(323):2(cid:26)(((cid:0)) (cid:1))pr(cid:23)otherwiseand the gradient is:r() =Prr(324):(cid:26)(cid:0)2(cid:1) ((cid:0))otherwise.(cid:23)pr(cid:23)0if ((cid:0)) (cid:1)0pr(cid:23) <As the number of joint anglesalong the chain grows, the computationalncomplexity ofandis linear for the listed goal types, since the end-e(cid:11)ectorGgmodule needs() time and the goal module needs(1) time.OnO3.3.3 Assembling Multiple ConstraintsIn the single constraint module, we consider only those joint angles whichlie on the constraint chain. To make our constraint system useful, multipleconstraints would be combined to one constraint, as in equations (3.1) or(3.3). The arguments in these formulas, the vector, may involve joint angles(cid:18)along many paths in the (cid:12)gure tree.In other words, with respect to this92CHAPTER 3. SPATIAL INTERACTION(cid:18)Githe gradient ofin (3.1) or (3.3) contains many zeros. For the sake ofcomputational e(cid:14)ciency and program modularity, we choose not to pass theoverall index of joint angles to the single constraint module.Suppose there areconstraints. Theth constraint involvesjointminiangles, whereas the combined constraint involvesjoint angles. Notice thatnthisis not simply the summation ofs, since one joint angle may benniinvolved in several chains. The relationship between the index used in thethiconstraint chain and the overall index is represented by the mappingM;; : : : ; n;; : : : ; n: f12g (cid:0)! f12g ;ii(325):theth joint angle in theth constraint chain corresponds to the()th jointjiMjiangle in the overall index system.Letdenote the vector formed by the joint angles along theth constrainti(cid:18)ichain. For disjunctively combined goals withde(cid:12)ned in (3.3) , one canGalways focus on the goal with minimal) at each iteration. So this isG(cid:18)i(inothing but a collection of goals.This module is designed for conjunctively combined goals withde(cid:12)nedGin (3.1). The evaluation ofis simply a summation of the individual. ToGGicompute the overall gradientggg= ((cid:1) (cid:1) (cid:1))g12nT= r(cid:18)G(cid:18)()by using the outputs of the single constraint moduleg(cid:18)i(i) = ((cid:1) (cid:1) (cid:1))gggiii12niT= r(cid:18)i(i) ,iG(cid:18)we only need to doj1.0, for= 12gj;; : : : ; n2. For= 1 todoimMjMjii()()jiigg+, for= 12wgj;; : : : ; ni.From above one can see that all theand their gradientscan beGigicomputed in parallel.We assumed in Section 3.3.2 that the constraint chain went from the rootof the (cid:12)gure tree to the distal segment. It is possible and sometimes usefulthat the chain go from a speci(cid:12)ed joint which is nearer to the root than thedistal segment is. Then we must take care of those joints which a(cid:11)ect an end-e(cid:11)ector, but are not in the constraint chain. For example, suppose that thetorso is the root of the (cid:12)gure, one constraint chain is from the right shoulderto the right hand, and the other is from the torso to the left hand. Althoughthe torso is not assigned to the end-e(cid:11)ector right hand, it will a(cid:11)ect the righthand when it moves the left one. The system should add joints from the torsoto the left shoulder to the constraint chain for the right hand [BOK80].3.3.INVERSE KINEMATIC POSITIONING93Constraints may exist in a hierarchy. With multiple constraints, there maybe one (or more) constraint(s) which consists of several constraints disjunc-tively combined. In general, a logic formula which contains conjunctive anddisjunctive connectors can be implemented. For example, the exterior of aconvex polyhedron can be de(cid:12)ned as the union of the outside half-space of allof its surrounding faces. This can be used to avoid obstacles: we may wantthe hand to get somewhere while keeping the elbow away from the obstacle.This does not solveobstacle avoidance, however.segmentJoints can be locked or added to the constraint chain for a particular task.3.3.4 Sti(cid:11)ness of Individual Degrees of FreedomA nonlinear programming algorithm which utilizes gradient information has aproperty that, given the same termination tolerance, a variable would undergomore displacement if the value of the ob jective function changes more due tothe unit increment of that variable. This property can be used to control thesti(cid:11)ness of each DOF by assigning a scaling factor to each joint angle. Thesescaling factors change the unit of each joint angle.For example, if the standard unit of joint angles is a radian, and 0.5 isassigned to a particular joint angle, which is to take half of the radian asthe unit of that joint angle, then this joint angle will be more reluctant tomove. One unit of change in this joint angle will have half of the e(cid:11)ect on theend-e(cid:11)ector as it would without that scaling factor. This makes the e(cid:11)ect ofother joint angles more apparent.3.3.5 An ExampleJackcontains an implementation of this multiple spatial constraint solver as abasic tool for the interactive manipulation of articulated (cid:12)gures. Constraintscan be of any types listed above, or can be sets of simple constraints disjunc-tively combined. Users may create various constraints, any of which can bemoved interactively, and the achieving con(cid:12)guration is solved and observed inreal-time.The pose in Figure 3.9 is achieved by using 6 constraints. Two Posi-tion/Orientation goals are used for two hands to hold the tube, where onedirection of the orientation is suppressed so that only normals of the handand the tube are aligned. Two Plane goals are used to constrain two elbowson two side planes. To have the person look down towards the end of thetube, we used two goals { a Line goal was used to constrain the view pointon the central axis of the tube; an Aiming-at goal was used to point the viewvector towards the end of the tube.In all, 22 DOFs are involved. To en-courage forward bending, we set the rigidities of the lateral bending and axialrotation of the torso to a mid-range (0.5) value. Starting from an uprightneutral position and moving from the waist, the solution took only 2 secondson a Silicon Graphics Personal Iris (4D-25TG), not one of the faster machinesof its type.94CHAPTER 3. SPATIAL INTERACTIONFigure 3.9: Looking Down Towards the End of the Tube.3.4 Reachable Spaces2Theis the volume or space encompassing all points that a ref-workspaceerence point on the hand (or the end e(cid:11)ector) traces as all the joints movethrough their respective ranges of motion [KW81]. An articulated chain isa series of links connected with either revolute or prismatic joints such as arobotic manipulator or a human limb. Visualizing the 3D workspace for artic-ulated chains has applications in a variety of (cid:12)elds: in computer graphics andarti(cid:12)cial intelligence systems that generate plans for approaching and grasp-ing an ob ject in complex environments, in CAD systems to design the interiorlayout of cars, vehicles, or space shuttles, in the evolving areas of telerobotsand parallel manipulators, in the coordination of di(cid:11)erent manipulators toperform certain tasks, and, (cid:12)nally, in ergonomic studies to help understand-ing the e(cid:11)ects of body size, joint limits, and limb length on the workspacePrevious workspace algorithms [Kum80, Sug81, GR82, TS83, TS81, Tsa86,YL83, Vij85, Muj87] are only capable of displaying 2D workspace cross-volume.2Tarek Alameldin.3.4. REACHABLE SPACES95sections. This is not adequate for redundant (more than 6 DOFs) manip-ulators with joint limits. We describe how to compute the 3D workspace forredundant articulated chains with joint limits.The (cid:12)rst e(cid:11)orts to compute the manipulator workspace, based on its kine-matic geometry, started in the mid 1970’s [Rot75, Sug81]. The (cid:12)rst resultwas that the extreme distance line between a chosen point on the (cid:12)rst jointaxis and the center point of the hand/end e(cid:11)ector (extreme reach) intersectsall intermediate joint axes of rotation. This is not valid, however, if any jointhas limits, any intermediate joint axis is parallel to the extreme distance line,or two joint axes intersect.Kumar and Waldron [KW81] presented another algorithm for the manip-ulator’s workspace.In their analysis, an imaginary force is applied to thereference point at the end e(cid:11)ector in order to achieve the maximum extensionin the direction of the applied force. The manipulator reaches its maximumextension when the force’s line of action intersects all joint axes of rotation(since the moment of the force about each axis of rotation must be zero).Every joint of the manipulator can settle in either of two possible positionsunder the force action. Hence, this algorithm results in 2di(cid:11)erent setsn(cid:0)1of joint variables for a manipulator ofjoints in the direction of the appliednforce. Each set of joint variables results in a point on the workspace bound-ary. The concept of stable and unstable equilibrium is used to select the setof joint variables that result in the maximum extension in the force direction.This algorithm is used to generate a shell of points which lie on the workspaceboundary by varying the direction of the applied force over a unit sphere.This algorithm has exponential time complexity and deals only with thosemanipulators that have ideal revolute joints.Tsai and Soni [TS83] developed another algorithm to plot the contour ofthe workspace on an arbitrarily speci(cid:12)ed plane for a manipulator withrev-nolute joints. The robot hand is moved to the speci(cid:12)ed plane, then the tipof the hand is moved on the plane until it hits the workspace boundary, and(cid:12)nally the workspace boundary is traced by moving the hand from one posi-tion to its neighbor. Each of these three subproblems is formulated as a linearprogramming problem with some constraints and bounded variables (to ac-count for the joint limits). Accordingly, this algorithm is just a 2D workspacecross-section computation and, moreover, has excessive computational cost.Yang [YL83] and Lee [LY83] presented algorithms to detect the existenceof holes and voids in the manipulator’s workspace. A workspace is said tohave a hole if there exist at least one straight line which is surrounded bythe workspace yet without making contact with it. The hole in a donut isa simple example for the above de(cid:12)nition. A workspace is said to have avoid if there exist a closed region, buried within the reachable workspace,Rsuch that all points inside the bounding surface ofare not reached by theRmanipulator. Gupta [Gup86, GR82] classi(cid:12)ed voids into two di(cid:11)erent types.The (cid:12)rst one, called, occurs around the (cid:12)rst axis of rotation and is likecentralthe core of an apple. The second type, calledor, occurstoroidalnoncentralwithin the reachable workspace and is like a hollow ring. He [Gup86] also96CHAPTER 3. SPATIAL INTERACTIONpresented qualitative reasoning about the transformation of holes to voidsand vice versa. Both the qualitative method developed by Gupta [Gup86]and the analytical one developed by Yang and Lee [YL83, LY83] are based onmapping the workspace from the distal link to the proximal one and studyingthe relationship between the generated workspace and the new axis of rotation.Tsai [Tsa86] presented another algorithm, based on the theory of recipro-cal screws. In contrast to the above algorithms that only compute workspacepoints, this algorithm traces the 2D workspace boundary for a given manip-ulator. The use of reciprocal screw theory has made computing piecewisecontinuous boundary that consists of straight line segments and circular arcspossible. The manipulator’s workspace is computed by performing the unionoperation on all the workspaces of the manipulator’s aspects. An aspect of arobot is interpreted as a set of joint variables such that the manipulator canreach points inside the workspace at one con(cid:12)guration without hitting a jointlimit [Tsa86]. The computed workspace has interior surfaces which are theboundaries of aspects. This algorithm is limited to manipulators which donot have holes or voids in their workspaces.Korein [Kor85] created conservative approximations to 3D reach volumesby taking polyhedral unions of reach polyhedra, working along an articulatedchain from the distal joint inwards. The ma jor drawbacks of his approachare the high computational cost and numerical sensitivity of the polyhedralunions which are very di(cid:14)cult to perform once they become many-sided.3.4.1 Workspace Point Computation ModuleThe purpose of this module is to compute a suitably dense set of workspacepoints. The inputs to this module are a chain of linkages with a proximaland distal end, the joint limits associated with each DOF and the desiredresolutions in the x, y and z directions () for the end e(cid:11)ectorres; res; resxyzposition.follows:We classify the algorithms that can be used to implement this module as1. Algorithms based on forward kinematics. The basic idea is to generateend e(cid:11)ector positions by cycling each DOF through some number ofdiscrete angles (if revolute) or distances (if prismatic).2. Algorithms based on nonlinear programming. Here a collection of pointsin space is provided as targets for the end e(cid:11)ector and the linkage at-tempts to solve for a satisfying posture.3. Algorithms based on force application at the end e(cid:11)ector. A series offorce directions is used to pull the chain to its maximum extension.Each class is better than the others for some applications. Direct kine-matics algorithms lend themselves easily to volume visualization applicationssince they require less time and space than the other algorithms. It is di(cid:14)cult,3.4. REACHABLE SPACES97however, to determine the adequate density (the number of points to be gen-erated) that would compute a workspace with the given resolution (res; res;xyand) . Hence, direct kinematics based algorithms cannot be used alonereszto compute the workspace volume since they are not guaranteed to computeall the reachable points. The resolution values divide the space into cells ofdimensions. A cell is marked with one if it contains aresresresx(cid:2)y(cid:2)zworkspace point. A cell marked with zero does not necessarily mean thatit is unreachable since the direct kinematics based algorithm might not havecomputed enough workspace points. This limitation is serious especially if theapplication requires surface visualization which use edge detection algorithmas will be described in the next section. On the other hand, algorithms basedon nonlinear programming are more appropriate for applications that onlyrequire computing the envelope of the reachable workspace since these algo-rithms compute only points that lie on the workspace envelope. However, thecost of computing each point by the nonlinear programming based algorithmsis higher than the cost using forward kinematics based algorithms. Finally,nonlinear programming based algorithms are more appropriate for applica-tions that require partial surface computation in predetermined directions.They can also be used in volume visualization applications by dividing thespace into voxels (of dimension) and using the inverse kine-resresresx(cid:2)y(cid:2)zmatics algorithm to determine whether the cell is occupied or not. However,this operation is very costly and does not guarantee the correct result sincethe nonlinear programming algorithms do not necessarily return the globalmaximum or minimum (they might stop at local ones). Algorithms that arebased on force application are very costly since they require exponential timeto compute each point. All these algorithms can be hybridized to computebetter quality volumes or surfaces [Ala91, ABS90].3.4.2 Workspace VisualizationWe believe that either surface based techniques or voxel based techniquescan be used in workspace visualization depending on the application type. Ifthe application goal is to compute the workspace boundary in order to (cid:12)ndthe intersection with other psurf ob jects in the environment, surface basedtechniques can be used. On the other hand, binary voxel based techniqueslend themselves easily to applications that require computing cut-away views,changing the view point, (cid:12)nding the union or the intersection with other ob-jects that are represented by voxels, and trading o(cid:11) computation time againstThis module constructs a surface that encompasses the workspace pointsthat were computed by the workspace point generation module. We havedeveloped an algorithm that accepts the workspace contours computed bythe direct kinematics algorithm. The algorithm can be summarized in fourimage quality.steps:1. Region Filling. This step involves determining the number of regionsin a given workspace contour. The number of holes and voids in the98CHAPTER 3. SPATIAL INTERACTIONgiven workspace contour can be determined. Region (cid:12)lling algorithmsare a common graphics utility and are widely used in paint programs[Sha80, FB85, Fis90]. A region is a collection of pixels. There are twotypes of regions: 4-connected and 8-connected. A region is 4-connectedif adjacent pixels share a horizontal or vertical edge. A region is 8-connected if adjacent pixels share an edge or a corner. Region (cid:12)llingalgorithms start with a given seed point () and set this pixel and allx; yof its neighbors with the same pixel value to a new pixel value. A goodregion-(cid:12)lling algorithm is one that reads as few pixels as possible. Thealgorithm computes the number of regions in a given workspace cross-section (contour). We search the contour for a reachable workspacecell (marked with 1) and use it as a seed point. The set of all cellsconnected to the seed point comprise a reachable region. The region(cid:12)lling algorithm sets those cells to a new value that greater than 2. Theregion (cid:12)lling algorithm is called as many times as necessary in order toset di(cid:11)erent regions with unique values.2. Boundary Detection. The purpose of this step is to compute the bound-ary of di(cid:11)erent regions in a workspace cross-section. This is done bytesting the neighbors of each cell in the workspace cross-section. Anarray element of the workspace cross-section is considered a boundarycell if it has a di(cid:11)erent value from its neighbor.3. Contour Tracing. This step computes the edges that connect the bound-ary points for a given region.4. Triangulation. This step constructs the 3D workspace by tiling adjacentcontours with triangles. We have used the Fuchs’ algorithm [FKU77]that interpolates the triangular faces between parallel slices in order toconstruct the 3D workspace surface from the di(cid:11)erent cross sections.3.4.3 Criteria SelectionThis module interacts with the user and selects the most suitable point com-putation and visualization algorithms based on the application requirements.Those requirements (parameters) include:(cid:15). This parameter allows the user to select to either com-Surface/volumepute the workspace boundary (envelope) or compute the workspace vol-ume based on the application requirements.(cid:15). This parameter allows the user to compute either theComplete/partialfull 3D workspace or just the portion of interest. If the user selects apartial workspace, it then asks for either the bounding cube or spherethat limits the portion of interest. The criteria selection would call thenonlinear-programming based algorithms from the point computationmodule since they are most suitable for computing partial workspaces.3.4. REACHABLE SPACES99(cid:15). This parameter allows the user to select computingHoles and voidsholes and voids based on the application requirements. Computing theworkspace envelope without holes and voids is important for some ap-plications. On the other hand, computing the holes and voids is oftenunnecessary and anyway requires more time and space.(cid:15). This parameter allows the user to select between a simpleResolutionplot and a complex image based on the application deadline. The criteriaselection asks the user to enter values for the required resolution inx; y;zanddirections. These parameters are denotedandres; res;xyreszrespectively. The criteria selection passes those values to the pointcomputation algorithms so that they can compute the right number ofworkspace points. If the user is interested in a quick response regardlessof the image quality, low resolution values for theandres; res;resxyzparameters can be entered.We can now compute 3D workspaces for articulated chains with redun-dant DOFs and joint limits. The criteria selection module interacts with theuser and selects the most suitable workspace computation and visualizationalgorithms based on the application requirements. The second module of thatsystem computes workspace points for the given chain. The third module (cid:12)tsa 3D surface around the volume that encompasses the workspace points com-puted by the second module. An example of a 3D workspace for the left armof a seated (cid:12)gure is illustrated in Plates 4 and 5. In Plate 4 the global shape ofthe workspace is visible as a translucent surface surrounding the body, whilein Plate 5 we see what the (cid:12)gure can simultaneously reach with his left handandwithin the cockpit.see100CHAPTER 3. SPATIAL INTERACTIONChapter 4Behavioral ControlThe behaviors constitute a powerful vocabulary for postural control. Themanipulation commands provide the stimuli; the behaviors determine theresponse. The rationale for using behavioral animation is its economy ofdescription: a simple input from the user can generate very complex andrealistic motion. By de(cid:12)ning a simple set of rules for how ob jects behave,the user can control the ob jects through a much more intuitive and e(cid:14)cientlanguage because much of the motion is generated automatically.Several systems have used the notion of behaviors to describe and generatemotion [Zel91]. The most prominent of this work is by Craig Reynolds, whoused the notion of behavior models to generate animations of (cid:13)ocks of birdsand schools of (cid:12)sh [Rey87]. The individual birds and (cid:12)sh operate using asimple set of rules which tell them how to react to the movement of the neigh-boring animals and the features of the environment. Some global parametersalso guide the movement of the entire (cid:13)ock. William Reeves used the samebasic idea but applied it very small inanimate ob jects, and he dubbed theresult[Ree83].particle systemsBehaviors have also been applied to articulated (cid:12)gures. McKenna andZeltzer [MPZ90] describe a computational environment for simulating virtualactors, principally designed to simulate an insect (a cockroach in particular)for animation purposes. Most of the action of the roach is in walking, and agait controller generates the walking motion. Re(cid:13)exes can modify the basicgait patterns. The stepping re(cid:13)ex triggers a leg to step if its upper bodyrotates beyond a certain angle. The load bearing re(cid:13)ex inhibits stepping ifthe leg is supporting the body. The over-reach re(cid:13)ex triggers a leg to move ifit becomes over-extended. The system uses inverse kinematics to position thelegs.controls bipedal locomotion in a similar fashion (Section 5), but forJacknow we focus on simpler though dramatically important postural behaviors.101102CHAPTER 4. BEHAVIORAL CONTROL????Get info from mouse or keyboard- geometric movement?- command to execute?Execute Behavior FunctionsEvaluate ConstraintsRedraw Graphics WindowsFigure 4.1: The Interactive System Architecture.4.1 An Interactive System for Postural Con-trolThe human (cid:12)gure in its natural state has constraints on its toes, heels, knees,pelvis, center of mass, hands, elbows, head and eyes. They correspond looselyto the columns of the sta(cid:11) in Labanotation, which designate the di(cid:11)erentparts of the body.The heart of the interactive system is a control loop, shown in Figure 4.1.The system repeatedly evaluates the kinematic constraints and executes thebehavior functions[PB91]. It also polls the user for information, which canbe geometric movements through the direct manipulation operator describedin Section 3.1, or commands to execute which can change the state of thesystem or the parameters of the constraints. The behavior functions can alsomodify the state of the environment or the parameters of the constraints,as described below. Each iteration of the control loop is a time step in asimulated movement process, although this is an imaginary sense of time.There are four categories of controls for human (cid:12)gures, illustrated in Ta-ble 4.1. First, there are. These are the parameters ofbehavioral parametersthe body constraints and govern things like whether the feet should remainplanted on the (cid:13)oor or whether they should be allowed to twist as the bodymoves. Second, there are. These behaviors express relation-passive behaviorsships between di(cid:11)erent parts of the (cid:12)gure. These relationships are usuallymore complex than can be expressed through the behavioral parameters. Anexample of a passive behavior is the parametrization of the distribution of4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL103weight between the feet. Third, there are. Active behaviorsactive behaviorshave a temporal component. They wait for events or conditions to occur andthen (cid:12)re o(cid:11) a response which lasts for a speci(cid:12)ed duration. An example of anactive behavior is the automatic stepping action the (cid:12)gure takes just beforeit loses its balance. Finally, there are. These aremanipulation primitivesthe commands which allow the user to interactively drag or twist parts of a(cid:12)gure. These are the principal sources of input, the stimuli for the posturaladjustments, although much of the movement during a postural adjustmentusually comes from the response generated by the behavioral controls. Ta-ble 4.1 provides a summary of’s current vocabulary for postural control.Jack4.1.1 Behavioral ParametersThe behavioral parameters are the properties of the constraints which modelthe posture. Mostly, these parameters describe the ob jective functions of theconstraints, as in whether the constraint speci(cid:12)es position, orientation, orboth. The parameters can include the goal values of the constraints as well.These are simple relationships which require no computation on the part ofthe system.Thebehaviors provide control over the position and orientation of theJackfeet, the elevation of the center of mass, the global orientation of the torso, theorientation of the pelvis, and the gaze direction of the head and eyes. Thereare additional non-spatial controls on the knees and elbows. These controlswere originally designed for the human (cid:12)gure in a rather ad hoc fashion,although they correspond loosely to the columns of the sta(cid:11) in Labanotation.Most of’s current behavioral parameter commands, listed in Table 4.1,Jackallow the user to instruct the (cid:12)gure to maintain the current posture of thepart of the body that it controls, such as the position and orientation offeet or the elevation of the center of mass. This means that this propertywill be maintained whenever possible even as the other parts of the (cid:12)gurechange posture. This works well in an interactive manipulation context: theuser manipulates the body part into place, and it stays there. This is the\what you see is what you get" approach. This also obviates the need for acomplex syntax for describing positions and orientations grammatically, sincethe method of describing the location is graphical and interactive, throughdirect manipulation.The Position and Orientation of the FeetThe foot behaviors are shown in Table 4.2. These are options to theset footbehaviorcommand. The standard human (cid:12)gure model inhas a foot withJacktwo segments connected by a single toe joint, and two natural constraints, oneon the toes and another on the heel. The toe constraint keeps the toes on the(cid:13)oor; the heel constraint can keep the heel on the (cid:13)oor or allow it to rise ifnecessary. For a standing (cid:12)gure, the pair of behaviorsandkeep heel on (cid:13)oorallow heel to risepivotcontrol the height of the heel. Thebehavior instructs104CHAPTER 4. BEHAVIORAL CONTROLbehavioral parametersset foot behaviorpivothold global locationhold local locationkeep heel on (cid:13)oorallow heel to riseset torso behaviorkeep verticalhold global orientationset head behavior(cid:12)xate head(cid:12)xate eyesrelease headrelease eyesset hand behaviorhands on hipshands on kneeshold global locationhold local locationrelease handshand on sitegrab ob jectpassive behaviorsbalance point follows feetfoot orientation follows balance linepelvis follows feet orientationhands maintain consistent orientationroot through center of massactive behaviorstake step when losing balancetake step when pelvis is twistedmanipulation primitivesmove footmove center of massbend torsorotate pelvismove handmove headmove eyesTable 4.1: Behavioral Controls.4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL105set foot behaviorpivothold global locationhold local locationkeep heel on (cid:13)oorallow heel to riseTable 4.2: Foot Behaviors.the toes to maintain the same position, and to maintain an orientation (cid:13)at onthe (cid:13)oor while allowing them to rotate through a vertical axis. Thehold globallocationpivotbehavior disables thebehavior and (cid:12)xes the toe orientation inspace. This is the appropriate behavior when the foot is not on the (cid:13)oor.Thebehavior attaches the foot to an ob ject such as a pedal.hold local locationIf the ob ject moves, the foot will follow it and maintain the same relativedisplacement from it. If the (cid:12)gure is seated, then the heel behaviors and thepivotholdbehavior have no e(cid:11)ect, and thebehaviors control the position andorientation of the heel instead of the toes.The behavior of the feet is usually activated by the manipulation of someother part of the (cid:12)gure, such as the center of mass or the pelvis. A goodexample of thebehavior is when the center of mass is dragged towardspivotone foot: should the other foot pivot in order to extend the leg, or should itremain planted and inhibit the movement of the center of mass? The behaviorssay which should occur.The Elevation of the Center of MassThe horizontal location of the center of mass is a passive behavior whichdetermines balance, as described below. The elevation of the center of massis more straightforward. This concept has a direct analog in Labanotation:the[Hut70]. A middle level of support is a natural standinglevel of supportposture, a low level of support is a squat, and a high level of support isstanding on the tip-toes. Thebehavior is an optionhold current elevationof thecommand. It instructs the (cid:12)gure to maintain theset balance behaviorcurrent elevation of the center of mass. This behavior is o(cid:11) by default. Thisbehavior is necessary because under normal circumstances, the center of massof the (cid:12)gure is free to rise and fall as necessary to meet the requirements of thefeet. After adjusting the center of mass to an appropriate level, if no controlholds it there, it may rise or fall unintentionally.The Global Orientation of the TorsoThe torso behaviors are listed in Table 4.3. The default behavior iskeepvertical, which causes the torso to maintain a vertical orientation. Biomechan-ics research tells us that one of the most constant elements in simple human106CHAPTER 4. BEHAVIORAL CONTROLset torso behaviorkeep verticalhold global orientationTable 4.3: Torso Behaviors.locomotor tasks is the global orientation of the head. One theory explainingthis suggests that the head is the principle sensor of stability [BP88]. Thekeep verticalbehavior mimics this nicely through a directional constraint onthe chest to remain vertical, while not a(cid:11)ecting its vertical rotation. Thismeans that as the pelvis of the (cid:12)gure rotates forward, backward, or side toside, the torso will automatically compensate to keep the head up. Since theconstraint is on the upper torso, not the head, the neck is free to move in orderfor the (cid:12)gure to look at certain reference points, as described below with thehead and eye behaviors.Thebehavior involves all three DOFs of the torso.hold global orientationThis allows other parts of the body to be adjusted while the head and cheststay relatively (cid:12)xed. This is particularly important in making adjustments tothe pelvis and legs after positioning the torso acceptably. This behavior doesnot involve position, because it is usually acceptable to have the position (cid:13)oatwith the rest of the body.Movements of the spine are described in terms of total bending anglesin the forward, lateral, and axial directions. The technique uses weightingfactors that distribute the total bending angle to the individual vertebrae insuch a way that respects the proper coupling between the joints. Di(cid:11)erentweight distributions generate bends of di(cid:11)erent (cid:13)avors, such as neck curls orbends con(cid:12)ned to the lower back. These parameters are options to the torsobehavior controls through thecommand because they governset torso behaviorhow the torso behaves as it bends to maintain the proper orientation. Theuser can select one of the standardoroptions,curl from neckbend from waistor alternatively input the range of motion of the spine by selecting a top andbottom joint, andandjoints which control the weightinginitiatorresistorbetween the vertebrae.The Fixation Point for the Head and EyesThe head and eyes can be controlled by specifying a (cid:12)xation point, modeledthrough aiming constraints which orient them in the proper direction. Theconstraint on the head operates on a reference point between the eyes, orientedforwards of the head. The head constraint positions only the head, using theneck. The constraint on the eyes rotates only the eyeballs. The eyeballs rotateside to side and up and down in their sockets. The behavioral parameterscontrol the head and eyes independently during postural adjustments. Theactive behaviors described below simulate the coupling between head and eyemovement.4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL107set head behavior(cid:12)xate head(cid:12)xate eyesrelease headrelease eyesTable 4.4: Head Behaviors.set hand behaviorhands on hipshands on kneeshold global locationhold local locationrelease handshand on sitegrab ob jectTable 4.5: Hand Behaviors.Thebehavior option of thecommand allows(cid:12)xate headset head behaviorthe user to select a (cid:12)xation point for the head. Thebehavior does(cid:12)xate eyesthe same for the eyes. When these behaviors are active, the head and eyeswill automatically adjust to remain focused on the (cid:12)xation point as the bodymoves.The Position and Orientation of the HandsThe principal control for the hands involves holding them at particularpoints in space as the body moves. Postural control of the arms and hands isusually a two step process. First, get the hands into position using the activemanipulation facilities, and second, set a control to keep them there as someother part of the body moves. Theandhold global locationhold local locationbehaviors serve much the same for the hands as their counterparts for the feet.The desired geometric positions and orientations are either global or local tosome other ob ject. Thecommand provides several standardset hand behaviorpostures. For a standing (cid:12)gure, a pleasing reference point is the (cid:12)gure’s hips.The hands rest on the hips with the elbows out to the side, the arms akimboposture. For a seated position, a pleasing reference point is the (cid:12)gure’s knees.Thebehavior moves the hand to a particular site, in both position andsiteorientation. This simulates a reaching movement, but its real purpose is tohold the hand there once it reaches the site.The controls for the hands are invoked whenever the body moves or when-ever an ob ject to which the hand is constrained changes location. A good108CHAPTER 4. BEHAVIORAL CONTROLanalogy is holding onto an ob ject such as a doorknob. If the door closes, thearm goes with it. Likewise, if the body bends over, the door stays (cid:12)xed andthe arm adjusts accordingly.Jackalso allows the converse relationship which is more suited to the waya person holds a screwdriver. A screwdriver is controlled completely by thehand and is not (cid:12)xed in space in the same sense as the door. If the bodybends over, the screwdriver should move along with the arm, not remain inplace like the doorknob. This type of a relationship comes from thegrab ob jectbehavior. \Grab" in this context does not mean \grasp"; it doesn’t mean the(cid:12)ngers will wrap around the ob ject. Such an action is available as a type ofmotion (Section 4.2.7). It actually means that the ob ject will subsequentlybe attached to the (cid:12)gure’s hand, as if the (cid:12)gure grabbed it. Once again, theprocess has two-steps: (cid:12)rst, position the hand and the ob ject relative to eachother, then specify thebehavior to hold it there.grabThe constraints on the hands are logically separate from the other con-straints on the body.evaluates the hand constraintsthe otherJackafterconstraints, not simultaneously. The reasons for structuring the arm behav-iors this way are partly practical and partly philosophical.In practice, the inverse kinematics algorithm does not perform well whenthe hand constraint is considered collectively with the other body constraints.With the human (cid:12)gure rooted through the toes, there are too many DOFsbetween the toes and the hands to be controlled e(cid:11)ectively. Even though theconstraints on the pelvis, center of mass, and opposite foot help to resolvethis redundancy, if the hand constraint is on equal par with the other parts ofthe body, the hand constraint can frequently cause the other constraints to bepulled away from their goals. Since the potential energy function describingthe equilibrium state for the (cid:12)gure is a weighted combination of all of theconstraints, the center of mass and pelvis constraints must have signi(cid:12)cantlyhigher weight to avoid having the hand pull the body o(cid:11) balance. It has beendi(cid:14)cult to arrive at a set of weights which give the right behavior. It has beenmuch easier to simply localize the movements of the arms and isolate themfrom the rest of the body.Philosophically, it is acceptable to consider the arm movements indepen-dently as well. Normally, a reaching task does not initiate much movement ofthe lower body, unless there are explicit instructions to do so. For example,consider what happens when a human being reaches for a nearby ob ject suchas a doorknob. If the door is near enough, this won’t involve any bendingof the waist, but if the door is farther away, it may be necessary to bendthe waist. If the door is farther away still, it may be necessary to squat orcounter-balance by raising a leg backwards. This system of behaviors requiresan explicit control specifying which approach the (cid:12)gure takes. If the torsomust bend or the center of mass must shift in order to perform a reachingtask, then the user, or some higher level behavior function, must initiate it.Automatic generation of these intermediate postures is non-trivial. We discusstwo rather di(cid:11)erent approaches in Sections 5.4 and 5.5.4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL109The Knees and ElbowsThe knees and elbows require special care to prevent them from becominglocked at full extension. The fully extended position not only appears awk-ward, but it tends to cause the inverse kinematics algorithm to get trappedin a local minimum. Because the algorithm uses a gradient descent approach,if an elbow or knee reaches its limit, it has a tendency to stay there. To1prevent this,usesconstraints to discourage the knees andJacklimit springelbows from reaching their limiting value. The springs give a high energy levelto the fully extended angle. The springs can be tuned to any angle, but thedefault is 10, and in practice, this tends to work well.(cid:14)This gives the (cid:12)gure in its natural standing position a more pleasing pos-ture, more like \at ease" than \attention". Biomechanics literature describesthis in terms of the stresses on the muscles [Car72]. Labanotation uses thisposture as the default: the elbows are \neither bent nor stretched" [Hut70]and, in the middle level of support, the knees are \straight but not taut"[Hut70].The PelvisThe pelvis and torso are intricatedly related. The torso includes all jointsin the spine from the waist to the neck, and rotating these joints allows the(cid:12)gure to bend over. However, when human beings bend over, they generallybend their pelvis as well as their torso. This means manipulating the two hipjoints as a unit, which can be a problem for a computer model because thereis no single (cid:12)xed point below the hips from which to rotate. However, thisproblem is easy to handle by controlling the orientation of the pelvis througha constraint.4.1.2 Passive BehaviorsPassive behaviors can represent more complex relationships than the behav-ioral parameters. They are like little processes attached to each (cid:12)gure. Thepassive behavior functions are executed at each interactive iteration. A pas-sive behavior can involve a global property of the (cid:12)gure such as the center ofmass or the shape of the (cid:12)gure’s support polygon. An example of this kindof behavior is the parametrization of the distribution of the weight betweenthe (cid:12)gure’s feet: when the feet move, the balance behavior function mustcompute the proper location for the balance point and register this with theconstraint on the center of mass. Passive behaviors arein thatinstantaneousthey explicitly de(cid:12)ne a relationship to be computed at each iteration.Passive behaviors are easy to implement in this basic system architecturebecause their only job is to compute the necessary global information and sup-ply it to the behavioral controls. Because of the general nature of the inverse1Essentially the algorithm sees a zero gradient and hence (cid:12)nds no advantage to movingthe locked joint as that does not decrease the overall distance to the goal: the requiredmotion is in fact exactly perpendicular to the aligned segments.110CHAPTER 4. BEHAVIORAL CONTROLkinematics constraints, the behaviors can overlap to a degree not possible withother systems, like Zeltzer’s local motion processes [Zel82, Zel84].Currently,has implemented six basic passive behavior functions forJackhuman (cid:12)gures, and they illustrate a range of capabilities. They control thelocation of the balance point, the orientation of the feet, the orientation ofthe pelvis, and the orientation of the hands. The (cid:12)nal two behaviors controlthe (cid:12)gure root.Balance as a Passive BehaviorProbably the most important human postural behavior, and the one demand-ing the most coordination, is balance. The need to remain balanced dictatesmuch of the subtle and elusive behavior of a human (cid:12)gure. The location ofthe balance point of a (cid:12)gure is signi(cid:12)cant in both cause and e(cid:11)ect. The lo-cation of the balance point is dependent on other parts of the (cid:12)gure, namely,the feet. Also, the balance point sends information to the other parts of thebody regarding the (cid:12)gure’s state of balance. Requiring a process to handlebalance in a global fashion was recognized long ago [BS79, BOK80], but sig-ni(cid:12)cant progress in computer interactivity and posture behavior algorithmswas needed to realize that design.To parametrize the location of the balance point with respect to the feetwe use the, which is the line between a (cid:12)xed reference point inbalance linethe middle of each foot. Biomechanics literature [Car72] states that in thestanding rest position, the body’s vertical line passes 2-5cm in front of theankle joint, midway through the arch of the foot. This line between the feetdivides the support polygon down the middle.Given the location of the center of mass, the balance point parameters,call themand, can be determined as shown in Figure 4.2. To do this,xzpro ject the balance point on the= 0 plane and call the point. Thenyb(cid:12)nd the point on the balance line closest to this point, and call itp.zisthe distance betweenand, that is, the balance point’s distance forwardbpfrom the balance line. However, it is more convenient to normalizebetweenz0.0 and 1.0 according to the placement ofbetween the balance line andbthe front edge of the support polygon. Therefore, if1 then the balancez >point lies outside the support polygon.If the balance point is behind thebalance line, then letbe normalized between -1.0 and 0.0. Likewise,is thezxinterpolation factor which givesin terms of the left and right foot referenceppoints, normalized between 0.0 and 1.0, with= 0 being the left foot. Ifxxis outside of the [01], then the balance point is to the side of the support;polygon.Once the system has the ability to measure balance, these parametersare available for the behavior functions to use. Thebalance point follows feetbehavior, described in Section 4.1.2, falls directly out of this parametrization.This behavior causes the distribution of weight between the feet to remainconstant even as the feet move. The active stepping behaviortake step whenlosing balance, described in Section 4.1.3, uses this parameter as its trigger.4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL111xbpzFigure 4.2: The Parametrization of the Balance Point.Global E(cid:11)ects of Local ManipulationsAnother capability of the passive behaviors in this system is to telegraphchanges in the posture of a local part of the (cid:12)gure to the rest of the (cid:12)gureas a whole. This can provide coordination between the di(cid:11)erent parts ofthe (cid:12)gure. The behavioral parameters as described above generally hold thedi(cid:11)erent parts of the (cid:12)gure in place, but sometimes it is better to have themmove automatically. A good example of this is thepelvis follows foot orientationbehavior, described in Section 4.1.2, in which the orientation of the pelvisautomatically adjusts to the orientation of the feet. Whenever the feet changeorientation, they radiate the change to the pelvis which mimics the rotationalspring-like behavior of the legs.Negotiating Position and OrientationThe passive behaviors o(cid:11)er a solution to the problem of negotiating the over-lapping in(cid:13)uence of position and orientation while interactively dragging partof the body. Because of the nature of the direct manipulation technique de-scribed in Section 3.1, it is not possible to rotate and translate during a singlemovement of the mouse. This has come up before, in Section 3.2.5: eitherthe dragging procedure hascontrol over orientation, in which case the ori-noentation is arbitrary and unpredictable, or the dragging procedurehavedoescontrol over orientation, in which case the orientation remains globally (cid:12)xedduring spurts of translation. Fixing the orientation during translation can, forexample, cause the hand to assume an awkward orientation as it is translated.Passive behavior functions allow the direct manipulation operator to havecontrol over the orientation and avoid awkward orientations. While the useris translating with the mouse, the behavior function can automatically deter-mine a suitable orientation based on heuristic observations. While rotating,the user has complete control over the orientation. The heuristics can simply112CHAPTER 4. BEHAVIORAL CONTROLbe embedded in the behavior functions (Section 2.4).The pair of behaviorsandfoot orientation follows balance linehands maintainconsistent orientation, use heuristics taken from Labanotation to predict suit-able orientations for the hands and the feet during their manipulation. Thisallows the user to position them mostly by translating them, making changesto the orientation only as necessary.The Figure RootOne passive behavior deserves special attention: the (cid:12)gure root. The prin-cipal disadvantage of modeling an articulated (cid:12)gure as a hierarchy is thatone point on the (cid:12)gure must be designated as the (cid:12)gure root. Section 3.2.3explains the e(cid:11)ect of the (cid:12)gure root on the inverse kinematics algorithm: thepositioning algorithm itself cannot move the (cid:12)gure root. It can only manipu-late chains emanating from the root. Any movement of the (cid:12)gure root mustbe programmed explicitly. Therefore, a ma jor element ofis thePeabodyability to change the setting of the (cid:12)gure root when necessary.The (cid:12)gure \root" is an unnatural concept. It has no natural analog fora mobile (cid:12)gure like a human being, so it has no place in the language forcontrolling human (cid:12)gures. Since it is a necessary internal concept, can it becontrolled internally as well? For certain postures of a human (cid:12)gure, thereare distinct reference points on the (cid:12)gure which serve as good (cid:12)gure roots:the feet, the lower torso, and the center of mass. It should be possible to havethe system choose these automatically and thus make the root transparent tothe user.There are several possibilities for the (cid:12)gure root of a human (cid:12)gure. Manysystems which don’t have the ability to change the root choose to locate it atthe lower torso [BKK85]. However, this complicates the process of moving+the lower torso during balance adjustments. Using this approach, it can bevery di(cid:14)cult to get the (cid:12)gure to bend over convincingly because the hipsneed to shift backwards and downwards in ways that are di(cid:14)cult to predict.However, for a seated posture, the lower torso is a good choice for the root.When a (cid:12)gure is standing, the feet are natural choices for the root.The choice of the (cid:12)gure root can be handled by designing a behavior func-tion which monitors the (cid:12)gure’s posture and automatically changes the (cid:12)gureroot when necessary to provide the best behavior. This behavior functionuses the following rules:(cid:15) It roots the (cid:12)gure through a foot whenever the weight of the body ismore than 60% on that foot. This ensures that if the (cid:12)gure is standingwith more weight on one leg than the other, the supporting leg servesas the root. It also ensures that if the (cid:12)gure is standing with weightequally between the two legs but possibly swaying from side to side thatthe root doesn’t rapidly vacillate between the legs.(cid:15) If the height of the center of mass above the feet dips below 70% ofthe length of the leg, then the root changes to the lower torso. This4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL113predicts that the (cid:12)gure is sitting down. Heuristically, this proves to bea good choice even if the (cid:12)gure is only squatting, because the constrainton the non-support leg tends to behave badly when both knees are bentto their extremes.Balance Point Follows FeetLabanotation has a notion for the distribution of the weight between the feetand the shifting of the weight back and forth [Hut70]. This notion is well-de(cid:12)ned regardless of the position of the feet: after specifying the distributionof weight between the feet, this proportion should remain (cid:12)xed even if theplacement of the feet need adjustment during a postural manipulation. Thisis the job of thebehavior.balance point follows feetGiven these two parameters, a new balance point can be computed basedon any new position of the feet. Holding these parameters (cid:12)xed as the feetmove ensures that the balance point maintains the same relationship to thefeet, both laterally and in the forward/backward direction.Foot Orientation Follows Balance LineDuring the active manipulation of the feet with thecommand, themove footuser can intersperse translations and rotations of the feet, centered around thetoes. Since it is not possible to rotate and translate during a single movement,either the dragging procedure hascontrol over orientation, in which casenothe orientation is arbitrary and unpredictable, or the dragging proceduredoeshave control over orientation, in which case the orientation remains globally(cid:12)xed during spurts of translation. Thefoot orientation follows balance linebehavior o(cid:11)ers a convenient alternative.The solution which the behavior o(cid:11)ers is to predict the proper orientationof the foot based on the balance line and adjust the orientation automaticallyas the foot is translated with thecommand. The balance line, asmove footdescribed above, is an imaginary line between the middle of the feet. Actually,this rule (cid:12)xes the orientation of the foot with respect to the balance line. Asthe foot translates, the balance line changes, and the orientation of the footchanges to keep the same relative orientation. This behavior is particularlyappropriate when the (cid:12)gure is taking a step forward with the intention ofturning to the side.Pelvis Follows Feet OrientationThe muscles in the leg make the leg act like a rotational spring. The hipand ankle joints provide only a little more than 90of rotation in the leg(cid:14)around the vertical axis. This means that the orientation of the feet and theorientation of the pelvis are linked together.If the orientation of the feetare (cid:12)xed, the orientation of the pelvis is severely limited. What is more,the extreme limits of pelvis orientation place an uncomfortable twist on thelegs. If the legs are rotational springs, then the \middle" orientation of the114CHAPTER 4. BEHAVIORAL CONTROLpelvis can be determined by simply averaging the orientation of the feet. Thisseems to be in fact what happens when a person stands naturally: the pelvis isoriented to relieve stress on the legs. Thebehaviorpelvis follows feet orientationsimulates this.Hands Maintain Consistent OrientationThe same problem with the orientation of the feet during thecom-move footmand occurs with the hands with thecommand. In fact, the prob-move handlem is more intricate because the hands have a much greater range of move-ment than the feet. How is the orientation of the hand related to its position?How can this be determined automatically in order to predict reasonable pos-tures when moving the hands?Labanotation suggests an answer. Labanotation has a detailed system fordescribing arm postures and gestures, but what is most interesting here iswhat the notation doessay. To simplify the syntax, Labanotation hasnota standard set of orientations for the palms when the arms are in speci(cid:12)cpositions. Notations need be made only when the orientations di(cid:11)er fromthese defaults. The rules are [Hut70]:(cid:15) When the arms hang by the side of the body, the palms face in.(cid:15) When the arms are raised forward or upward, the palms face towardseach other.(cid:15) When the arms are raised outward to the side, the palms face forward.(cid:15) When the arms cross the body, the palms face backward.These rules are useful as defaults, but of course they do not dictate absolutebehavior. These rules govern the orientation of the hands when the usertranslates them from one area to another without specifying any orientationalchange. These rules only take e(cid:11)ect when the hand moves from one region toanother.Root Through Center of MassMost of the behaviors described so far are only appropriate for standing (cid:12)g-ures, which of course means that they are also only appropriate for earth-bound (cid:12)gures. But what about (cid:12)gures in zero-gravity space? This is actuallyquite easy to simulate by rooting the (cid:12)gure through the center of mass anddisabling all other behaviors. The one constant element of zero-gravity is thecenter of mass. When the (cid:12)gure is rooted through the center of mass, theglobal location of the center of mass remains (cid:12)xed as the (cid:12)gure moves.4.1.3 Active BehaviorsActive behaviors mimic re(cid:13)exive responses to certain conditions in the body.They can have temporal elements, so they can initiate movements of parts of4.1. AN INTERACTIVE SYSTEM FOR POSTURAL CONTROL115the body which last for a certain duration. These behaviors make use of theconcept of a. A motion primitive has a distinct durationmotion primitivein terms of interactive iterations, and it typically involves a constraint whichchanges over this time interval. An example of this is the stepping movementof the feet which is initiated when the (cid:12)gure’s center of mass leaves its sup-port area. The interactive system architecture maintains a list of currentlytriggered active behaviors, and it advances them at each iteration until theyare complete. The behaviors terminate themselves, so the duration can beexplicit in terms of a number of interactive iterations, or they can continueuntil a certain condition is met.Active behaviors are like motor programs, or[Kee82, Ros91,schemasSch82b, Sch82a]. Considerable physiological and psychological evidence sug-gests the existence of motor programs, which are preprogrammed motor re-sponse to certain conditions. The theory of schemas suggests that humansand animals have catalogs of preprogrammed motor responses that are (cid:12)redo(cid:11) to produce coordinated movements in the body. Schemas are parametrizedmotor programs which can be instantiated with di(cid:11)erent settings. For somemotor programs, there even seems to be very little feedback involved. Ev-idence of this comes from experiments which measure the excitation of themuscles of the arm during reaching exercises. The patterns of excitation re-main constant even if the movement of the hand is impeded [Ros91].The incorporation of active behaviors into the postural control processbegins to blur the distinction between motion and manipulation. The purposeof the behaviors is predictive: if the user drags the center of mass of a (cid:12)gureaway from the support polygon, this probably means that the desired posturehas the feet in a di(cid:11)erent location. The job of the active behavior is toanticipate this and hopefully perform the positioning task automatically.There are two active behaviors, both involving the placement of the feet.Theandbehaviorstake step when losing balancetake step when pelvis is twistedautomatically reposition the feet just before the (cid:12)gure loses its balance. Theyuse the balance point parameters described above as their triggers. The pur-pose of these behaviors is to predict a proper posture for the (cid:12)gure given thatits center of mass is leaving the support polygon.Active behaviors can be used to simulate movement even in the context ofpostural control. The entire process of interactive postural control can serve asa good approximation to motion anyway. The active behaviors provide a wayin which motion primitives can be incorporated into the interactive system.To do this more e(cid:11)ectively, the interactive system needs a more sophisticatednotion of time and timed events (Section 4.3).Take Step When Losing BalanceThis behavior (cid:12)res a stepping response to the loss of balance of a (cid:12)gure. Whenthis behavior is active, it monitors the parametrization of the balance pointof the (cid:12)gure as described with thebehavior. If thebalance point follows feetbalance point leaves the support polygon to the front or back, the behavior116CHAPTER 4. BEHAVIORAL CONTROLmove footmove center of massbend torsorotate pelvismove handmove headmove eyesTable 4.6: The Manipulation Primitives.moves the non-support foot forward or backward to compensate. The non-support foot in this case is the one which bears less weight. The behaviorcomputes the new foot location such that the current balance point will liein the middle of the new support polygon once the foot arrives there. If thebalance point leaves the support polygon to the side, the stepping motionmoves the support foot instead. In this case, the support foot is the only onewhich can be repositioned in order to maintain balance.Take Step When Pelvis Is TwistedThe discussion of thebehavior above described thepelvis follows feet orientationrelationship between the global orientations of the feet and pelvis, particularlyin terms of determining an orientation for the pelvis from the orientation ofthe feet. The opposite relationship is possible as well. Consider standing withyour feet slightly apart, and then begin to twist your body to the right. Afterabout 45of rotation, your legs will not be able to rotate any more. In order(cid:14)to continue rotating, you will be forced to take a step, a circular step witheither your left or right foot.Thebehavior mimics this. When it senses thattake step when pelvis twistedthe orientation of the pelvis is near its limit relative to the feet, it repositionsthe non-support foot in an arc in front of or behind the other foot, twisted(cid:14)90.4.2Interactive Manipulation With BehaviorsThis section discusses themanipulation primitives, but in the processJackit describes the entire manipulation process, including the e(cid:11)ect of all of theimplemented behaviors. The e(cid:11)ect of the manipulation commands cannot betreated in isolation. In fact, the very nature of the system of behaviors impliesthat nothing happens in isolation. This discussion serves as a good summaryof these techniques because these are the commands (Table 4.6) which theuser uses the most. These are the verbs in the postural control language.4.2.INTERACTIVE MANIPULATION WITH BEHAVIORS117The interactive postural control vocabulary includes manipulation prim-itives which allow the user to push, poke, and twist parts of the body, andbehavior controls which govern the body’s response. The manipulation com-mands are su(cid:14)ciently intuitive to provide good handles on the (cid:12)gure, and thebehavioral controls make the responses reasonable.The structure of the behaviors for human (cid:12)gures did not come out of amagic hat. The rationale behind the behaviors comes partially from biome-chanics and physiology literature, and partially from the semantics of move-ment notations such as Labanotation. Labanotation provides a good set ofdefault values for the behaviors because it incorporates so many assumptionsabout normal human movement.4.2.1 The FeetThe feet can be moved with the active manipulation command. Thismove footcommand allows the user to drag the foot interactively. This automaticallytransfers the support of the (cid:12)gure to the other foot, provided the (cid:12)gure isstanding. The control over the position of the feet is straightforward. Themanipulation operator also gives control over the orientation. However, whiletranslating the foot, its orientation depends upon the foot orientation behav-ior. The default behavior maintains a constant global orientation. Thefootorientation follows balance linebehavior causes the orientation of the foot toremain (cid:12)xed with respect to the balance line during translation. This meansthat if the foot goes forward, it automatically rotates as if the (cid:12)gure is turningtoward the direction of the stationary foot.Thecommand automatically causes a change in the balance pointmove footaccording to thebehavior, which is the default. Thisbalance point follows feetmeans that the distribution of weight between the feet will remain constant asthe foot moves. The location of the balance point within the support polygon,both side to side and forwards/backwards, will remain (cid:12)xed as the supportpolygon changes shape. This is evident in Figure 4.3. The balance point shiftsalong with the foot. If this behavior is disabled, the balance point will remain(cid:12)xed in space.Manipulating the feet also telegraphs a change to the pelvis according tothebehavior, which is the default. This meanspelvis follows foot orientationthat as the foot rotates, the pelvis automatically rotates as well. This keepsthe body turned in the direction of the feet.4.2.2 The Center of Mass and BalanceThecommand allows the user to interactively drag themove center of massbalance point of the (cid:12)gure, shifting its weight back and forth or forward andbackward. This command changes the parametrization of the balance point interms of the feet. If thebehavior is active, then whenbalance point follows feetthecommand terminates, the balance point will remain atmove center of massits new location relative to the support polygon.118CHAPTER 4. BEHAVIORAL CONTROLFigure 4.3: Moving the Left Foot, with Balance Point Following Feet.The location of the balance point has a great e(cid:11)ect on the feet. If the footbehavior is, then shifting the weight laterally back and forth will causepivotthe feet to twist back and forth as well. On the other hand, if the feet donot pivot, then they remain planted, possibly inhibiting the movement of thebalance point. In Figure 4.4, the feet are held in place, not pivoting.Thecommand also gives control over the elevationmove center of massof the center of mass. Normally, the elevation of the center of mass is notcontrolled explicitly, except through thebehavior optionhold current elevationto thecommand. Thecommand givesset balance behaviormove center of masscontrol over the elevation, so moving the center of mass up and down allowsthe (cid:12)gure to stand on its tip-toes or squat down. Figure 4.5 shows the centerof mass being lowered into a squatting posture. The constraint on the pelvisensures that the hips remain square and straight.The movement of the center of mass also tends to trigger the rootingbehavior. This is mostly transparent, but to the trained eye, it is apparent inthe movement of the feet. The support foot (the rooted one) is always verystationary.The manipulation of the center of mass is the main instigator of the ac-tive stepping behavior. While the stepping behavior is active, if the balancepoint reaches the perimeter of the support polygon, the feet are automaticallyrepositioned by the stepping behavior. Figure 4.6 illustrates the stepping be-havior as the center of mass is dragged forward. When this occurs, the visualimpression is of the (cid:12)gure being pulled and taking a step just to prevent afall; it does not look like the (cid:12)gure is deliberately trying to walk somewhere.(In Section 5.2 more purposeful stepping and walking behaviors are utilized.)4.2.INTERACTIVE MANIPULATION WITH BEHAVIORS119Figure 4.4: Shifting the Center of Mass.Figure 4.5: Lowering the Center of Mass.120CHAPTER 4. BEHAVIORAL CONTROLFigure 4.6: Taking a Step before Losing Balance.4.2.3 The TorsoThespine model provides a very important biomechanical feature forJacke(cid:11)ective human behavioral control. Each vertebra has a current position de-(cid:12)ned by the three joint angles relative to its proximal vertebra. Also de(cid:12)nedin the spinal database are joint rest positions and 6 joint limits for every joint.If each attribute is summed up for all joints, then 3D vectors are de(cid:12)ned forcurrent position, joint rest position, and two joint limits for the global spine.The target position { the 3D vector sum of (cid:12)nal joint positions { is supplied asan input parameter. Movement towards the target position is either bendingor unbending, meaning either towards the joint limits or towards the spine’srest position. Motion is de(cid:12)ned as an interpolation between the current po-sition and either the spine’s position of maximum limit, or the spine’s restposition.dimension.Three rotations are calculated independently and then merged into one.For example, a 3D orientation vector (e.g. (cid:13)ex 45 degrees, rotate axially 20degrees left, and lateral bend 15 degrees right) can be accomplished in onefunction with 3 loop iterations. It is assumed for the model that the maximumvertebral joint limit in one dimension will not a(cid:11)ect the joint limits of anotherThe spine’s rest position is included in the model, because it is a positionof high comfort and stability.If the spine is unbending in one dimensionof movement, it will move towards that position of highest comfort in thatrotational dimension. The input parameters listed in Section 2.3 determinehow much each vertebra bends as the spine moves. The three dimensions are4.2.INTERACTIVE MANIPULATION WITH BEHAVIORS121done separately, then combined for the (cid:12)nal posture.A participation vector is derived from the spine’s current position, targetposition, and maximum position. This global participation represents a 3Dvector of the ratio of spine movement to the maximum range of movement.Participation is used to calculate the joint weights.The following formulas are de(cid:12)ned in each of three DOFs. LetT arget= spine target positionC urrent= spine current positionM ax= spine sum of joint limitsRest= spine sum of joint rest positions.If the spine is bending, then the participationisPT arget (cid:0) C urrentP=:M ax (cid:0) C urrentOtherwise, the spine is unbending andT arget (cid:0) C urrentP=:Rest (cid:0) C urrentThe joint positions of the entire spine must sum up to the target position.To determine how much the joint participates, a set of weights is calculatedfor each joint. The participation weight is a function of the joint number,the initiator joint, and the global participation derived above. Also, a resis-tance weight is based on the resistor joint, degree of resistance, and globalparticipation. To calculate the weight for each joint, let:iij= joint positionlimiti= the joint limitresti= the rest positionipir= participation weight= resistance weight.If the spine is bending, thenwp(cid:1) r(cid:1)limit(cid:0) j=();iiiiiwhile if the spine is unbending,wp(cid:1) r(cid:1)rest(cid:0) j=():iiiii122CHAPTER 4. BEHAVIORAL CONTROLThe weights range from 0 to 1. A weight of% means that the movementkwill go% of the di(cid:11)erential between the current position and either the jointklimit (for bending) or the joint rest position (for unbending).To understand resistance, divide the spine into two regions split at theresistor joint. The region of higher activity contains the initiator. Label theseregionsand. The e(cid:11)ect of resistance is that joints in the resis-activeresistivetive region will resist participating in the movement speci(cid:12)ed by the parameterdegree of resistance. Also, joints inbetween the initiator and resistor will haveless activity depending on the degree of resistance.Resistance does not freeze any of the joints. Even at 100% resistance, theactive region will move until all joints reach their joint limits. Then, if thereis no other way to satisfy the target position, the resistive region will beginto participate.If the desired movement is from the current position to one of two maxi-mally bent positions, then the weights calculated should be 1.0 for each jointparticipating. The algorithm interpolates correctly to either maximally bentposition. It also interpolates correctly to the position of highest comfort. Tocalculate the position of each jointafter movement succeeds, let:iMT arget (cid:0) C urrent== incremental movement of the spine.= joint positionij(cid:3)ji = new joint positionT arget= spine target positionC urrent= spine current positionThenand it is easy to show thati =:jT arget(cid:3)M wijij=+;iwiP(cid:3)P(cid:3)M wji=(j+)iiwiPPM wi=j+iPwiPP=+C urrentMP=+C urrentM=T arget:wiwiPPThecommand positions the torso using forward kinematics,bend torsowithout relying on a dragging mechanism. It consists of potentiometers whichcontrol the total bending angle along the three DOFs. The command also4.2.INTERACTIVE MANIPULATION WITH BEHAVIORS123prompts for the (cid:13)avor of bending. These controls are the same as for thesettorso behaviorcommand described above. They include options which specifythe range of motion of the spine, de(cid:12)ned through a top and bottom joint,along withandjoints which control the weighting betweeninitiatorresistorthe vertebrae.Bending the torso tends to cause large movements of the center of mass, sothis process has a great e(cid:11)ect on the posture of the (cid:12)gure in general, particu-larly the legs. For example, if the (cid:12)gure bends forward, the hips automaticallyshift backwards so that the (cid:12)gure remains balanced. This is illustrated in Fig-ure 4.7.4.2.4 The PelvisThecommand changes the global orientation of the hips. Thisrotate pelviscan curl the hips forwards or backwards, tilt them laterally, or twist theentire body around the vertical axis. The manipulation of the pelvis alsoactivates the torso behavior in a pleasing way. Because of its central location,manipulations of the pelvis provide a powerful control over the general postureof a (cid:12)gure, especially when combined with the balance andtorsokeep verticalconstraints. If the torso is kept vertical while the pelvis curls underneath it,then the torso curls to compensate for the pelvis. This is shown in Figure 4.8.Thecommand can also trigger the active stepping behavior ifrotate pelvisthe orientation reaches an extreme angle relative to the feet.4.2.5 The Head and EyesTheandcommands manipulate the head and eyes, re-move headmove eyesspectively, by allowing the user to interactively move a (cid:12)xation point. Thehead and eyes both automatically adjust to aim toward the reference point.The head and eyes rotate as described in Section 4.1.1.4.2.6 The ArmsThe active manipulation of the arm allows the user to drag the arm aroundin space using the mechanism described in Section 3.2.5. These movementsutilize the shoulder complex as described in Section 2.4 so that the coupledjoints have a total of three DOFs. Figure 4.10 shows the left hand beingmoved forwards.Although it seems natural to drag this limb around from the palm or (cid:12)n-gertips, in practice this tends to yield too much movement in the wrist and thewrist frequently gets kinked. The twisting scheme helps, but the movementsto get the wrist straightened out can interfere with an acceptable position forthe arm. It is much more e(cid:11)ective to do the positioning in two steps, the (cid:12)rstpositioning the arm with the wrist (cid:12)xed, and the second rotating the handinto place. Therefore, our active manipulation command for the arms cancontrol the arm either from a reference point in the palm or from the lower124CHAPTER 4. BEHAVIORAL CONTROLFigure 4.7: Bending the Torso while Maintaining Balance.Figure 4.8: Rotating the Pelvis while Keeping the Torso Vertical.4.2.INTERACTIVE MANIPULATION WITH BEHAVIORS125Figure 4.9: Moving the Head.Figure 4.10: Moving the Hand.126CHAPTER 4. BEHAVIORAL CONTROLend of the lower arm, just above the wrist. This process may loosely simulatehow humans reach for ob jects, for there is evidence that reaching involvestwo overlapping phases, the (cid:12)rst a ballistic movement of the arm towards therequired position, and the second a correcting stage in which the orientationof the hand is (cid:12)ne-tuned [Ros91]. If the target for the hand is an actual grasp,then a specializedbehavior for grasping may be invoked which e(cid:11)ectivelyJackcombines these two steps.4.2.7 The Hands and GraspingJackcontains a fully articulated hand. A hand grasp capability makes somereaching tasks easier [RG91]. The grasp action requires a target ob ject anda grasp type. Thegrasp is purely kinematic. It is a considerable conve-Jacknience for the user, however, since it virtually obviates the need to individuallycontrol the 20 DOFs in each hand.For a grasp, the user speci(cid:12)es the target ob ject and a grip type. Theuser chooses between a prede(cid:12)ned grasp site on the target or a calculatedtransform to determine the grasp location. A distance o(cid:11)set is added to thesite to correctly position the palm center for the selected grip type. The handis preshaped to the correct starting pose for the grip type selected, then thepalm moves to the target site.The (cid:12)ve grip types implemented are the power, precision, disc, small disc,and tripod [Ibe87]. The grips di(cid:11)er in how the hand is readied and whereit is placed on or near the ob ject. Once these actions are performed, the(cid:12)ngers and thumb are just closed around the ob ject, using collision detectionon the bounding box volume of each digit segment to determine when to ceasemotion.4.3 The Animation Interface2Theanimation system is built around the concept of a, whichJackmotionis a change in a part of a (cid:12)gure over a speci(cid:12)c interval of time. A motion isa rather primitive notion. Typically, a complex animation consists of manydistinct motions, and several will overlap at each point in time. Motionsare created interactively through the commands on theand themotion menuhuman motion menu. There are commands for creating motions which controlthe placement of the feet, center of mass, hands, torso, arms, and head.Jackdisplays motions in an animation window. This window shows timeon a horizontal axis, with a description of the parts of each (cid:12)gure which aremoving arranged vertically. The time interval over which each motion is activeis shown as a segment of the time line. Each part of the body gets a di(cid:11)erenttrack. The description shows both the name of the (cid:12)gure and the name of thebody part which is moving. The time line itself displays motion attributesgraphically, such as velocity control and relative motion weights.2Paul Diefenbach.4.3. THE ANIMATION INTERFACE127The numbers along the bottom of the animation grid are the time line. Bydefault, the units of time are in seconds. When the animation window (cid:12)rstappears, it has a width of 3 seconds. This can be changed with the arrowsbelow the time line. The horizontal arrows scroll through time keeping thewidth of the window constant. The vertical arrows expand or shrink the widthof the window, in time units. The current animation time can be set eitherby pressing the middle mouse button in the animation window at the desiredtime and scrolling the time by moving the mouse or by entering the currenttime directly through the.goto timeMotions actually consist of three distinct phases, although this is hiddenfrom the user. The (cid:12)rst stage of a motion is the pre-action step. This stepoccurs at the starting time of the motion and prepares the (cid:12)gure for theimpending motion. The next stage is the actual motion function itself, whichoccurs at every time interval after the initial time up to the ending time,inclusive. At the ending time after the last incremental motion step, thepost-action is activated disassociating the (cid:12)gure from the motion. Because ofthe concurrent nature of the motions and the possibility of several motionsa(cid:11)ecting the behavior of one moving part, these three stages must occur ateach time interval in the following order: motion, post-action, pre-action.This allows all ending motions to (cid:12)nish before initializing any new motionsa(cid:11)ecting the same moving part.While the above description implies that body part motions are controlleddirectly, this is not the true behavior of the system. The animation systemdescribes postures through constraints, and the motions actually control theexistence and parameters of the constraints and behaviors which de(cid:12)ne thepostures. Each motion has a set of parameters associated with it which con-trol the behavior of the motion. These parameters are set upon creation ofthe motion and can be modi(cid:12)ed by pressing the right mouse button in the an-imation window while being positioned over the desired motion. This changesor deletes the motion, or turns the motion on or o(cid:11).Each motion is active over a speci(cid:12)c interval in time, delimited by astart-ing timeending timeand an. Each motion creation command prompts forvalues for each of these parameters. They may be entered numerically fromthe keyboard or by direct selection in the animation window. Existing timeintervals can be changed analogously. Delimiting times appear as vertical\ticks" in the animation window connected by a velocity line. Selecting theduration line enables time shifting of the entire motion.The yellow line drawn with each motion in the animation window illus-trates the motion’s. Each motion describes movement of aweight functionpart of the body through a kinematic constraint. The constraint is only ac-tive when the current time is between the motion’s starting time and endingtime. It is entirely possible to have two motions which a(cid:11)ect the same part ofthe body be active at the same time. The posture which the (cid:12)gure assumes isa weighted average of the postures described by the individual motions. Theweights of each constraint are described through the weight functions, whichcan be of several types:128CHAPTER 4. BEHAVIORAL CONTROLconstantThe weight does not change over the life of the con-increaseThe weight starts out at 0 and increases to is maximumstraint.at the end time.0 at the end time.decreaseThe weight starts out at its maximum and decreases toease in/ease outThe weight starts at 0, increases to its max-imum halfway through the life of the motion, and then de-creases to 0 again at the end time.The shape of the yellow line in the animation window illustrates the weightfunction. The units of the weight are not important. The line may be thoughtof as an icon describing the weight function.The green line drawn with each motion in the animation window representsthe velocity of the movement. The starting point for the motion comes fromthe current posture of the (cid:12)gure when the motion begins. The ending positionof the motion is de(cid:12)ned as a parameter of the motion and is speci(cid:12)ed whenthe motion is created. The speed of the end e(cid:11)ector along the path betweenthe starting and ending positions is controlled through the velocity function:constantConstant velocity over the life of the motion.increaseThe velocity starts out slow and increases over the lifedecreaseThe velocity starts out fast and decreases over the lifeof the motion.of the motion.ease in/ease outThe velocity starts slow, increases to its max-imum halfway through the life of the motion, and then de-creases to 0 again at the end time.The shape of the green line in the animation window illustrates the velocityfunction. The scale of the velocity is not important. This line can be thoughtof as an icon describing the velocity.4.4 Human Figure MotionsThe commands on thecreate timed body motions. Thesehuman motion menumotions may be combined to generate complex animation sequences. Takenindividually, each motion is rather uninteresting. The interplay between themotions must be considered when describing a complex movement. Thesemotions are also mostly sub ject to the behavioral constraints previously de-scribed.Each one of these commands operates on a human (cid:12)gure. If there is onlyone human (cid:12)gure present, these commands automatically know to use that(cid:12)gure.If there is more than one human (cid:12)gure, each command will begin4.4. HUMAN FIGURE MOTIONS129by requiring the selection of the (cid:12)gure. Each of these commands needs thestarting and ending time of the motion. Default or explicitly entered valuesmay be used. The motion may be repositioned in the animation window usingthe mouse.A motion is a movement of a part of the body from one place to another.The movement is speci(cid:12)ed in terms of the (cid:12)nal position and the parametersof how to get there. Theposition of the motion, however, is de(cid:12)nedinitialimplicitly in terms of where the part of the body is when the motion starts. Forexample, a sequence of movements for the feet are de(cid:12)ned with one motion foreach foot fall. Each motion serves to move the foot from its current position,wherever that may be, when the motion starts, to the (cid:12)nal position for thatmotion.4.4.1 Controlling Behaviors Over TimeWe have already seen how the posture behavior commands control the e(cid:11)ectof the human movement commands. Their e(cid:11)ect is permanent, in the sensethat behavior commands and constraints hold continuously over the course ofan animation. The \timed" behavior commands on thehuman behavior menuallow specifying controls over speci(cid:12)c intervals of time. These commands,create timed (cid:12)gure supportcreate timed balance controlcreate timed torso control,,,create time hand controlcreate time head control, andeach allow a speci(cid:12)c intervalof time as described in Section 4.3 just like the other motion commands. Thebehavior takes e(cid:11)ect at the starting time and ends with the ending time. Atthe ending time, the behavior parameter reverts to the value it had before themotion started.4.4.2 The Center of MassA movement of the center of mass can be created with thecreate center of massmotioncommand. This controls the balance point of the (cid:12)gure. There are twoways to position the center of mass. The (cid:12)rst option positions the balancepoint relative to the feet by requiring a (cid:13)oating point number between 0.0and 1.0 which describes the balance point as an interpolation between the left(0.0) and right (1.0) foot; thus 0.3 means a pointof the way from the left310foot to the right. Alternatively, one can specify that the (cid:12)gure is standingwith 30% of its weight on the right foot and 70% on the left.Theoption causes the center of mass to move to a speci(cid:12)cglobal locationpoint in space. Herewill allow the user to move the center of mass toJackits desired location using the same technique as with themove center of masscommand on the.human manipulation menuAfter choosing the positioning type and entering the appropriate parame-ters, several other parameters may be provided, including the weight functionand velocity. The weight of the motion is the maximum weight of the con-straint which controls the motion, sub ject to the weight function.130CHAPTER 4. BEHAVIORAL CONTROLThe behavior of thecommand depends on thecreate center of mass motionsetting of the (cid:12)gure support.It is best to support the (cid:12)gure through thefoot which is closest to the center of mass, which is the foot bearing most ofthe weight. This ensures that the supporting foot moves very little while theweight is on it.The e(cid:11)ect of the center of mass motion depends upon both the settingof the (cid:12)gure support at the time the motion occurs and when the motion iscreated. For predictable behavior, the two should be the same. For example,if a motion of the center of mass is to take place with the (cid:12)gure seated, thenthe (cid:12)gure should be seated when the motion is created.The support of the (cid:12)gure can be changed at a speci(cid:12)c moment with thecre-ate timed (cid:12)gure supportcommand. This command requires starting and endingtimes and the (cid:12)gure support, just like thecommand. Whenset (cid:12)gure supportthe motion’s ending time is reached, the support reverts to its previous value.4.4.3 The PelvisThe lower torso region of the body is controlled in two ways: through thecenter of mass and through the pelvis. The center of mass describes thelocation of the body. The pelvis constraint describes the orientation of thehips. The hips can rotate over time with the command.create pelvis motionThecommand allows the user to rotate the pelvis intocreate pelvis motionthe (cid:12)nal position, using the same technique as thecommand. Itrotate pelvisalso requires the velocity, and weight functions, and the overall weight.4.4.4 The TorsoThe movement of the torso of a (cid:12)gure may be speci(cid:12)ed with thecreate torsomotion. This command permits bending the torso into the desired posture,using the same technique as thecommand. Like themove torsomove torsocommand, it also prompts for the torso parameters.Thecommand requires a velocity function, but not acreate torso motionweight or a weight function because this command does not use a constraintto do the positioning. Because of this, it is not allowable to have overlappingtorso motions.After the termination of a torso motion, the vertical torso behavior isturned o(cid:11). The behavior of the torso can be changed at a speci(cid:12)c momentwith thecommand. This command requires startingcreate timed torso controltime and ending times and the type of control, just like theset torso controlcommand. When the motion’s ending time is reached, the behavior revertsto its previous value.4.4.5 The FeetThe (cid:12)gure’s feet are controlled through the pair of commandscreate foot motionand. These two commands can be used in conjunction tocreate heel motion4.4. HUMAN FIGURE MOTIONS131cause the (cid:12)gure to take steps. The feet are controlled through constraintson the heels and on the toes. The toe constraints control the position andorientation of the toes. The heel constraint controls only the height of theheel from the (cid:13)oor. The position of the heel, and the entire foot, comes fromthe toes. The commands allow the selection of the right or left foot.Thecommand gets the ending position for the foot by thecreate foot motiontechnique of thecommand. In addition, a height may be speci(cid:12)ed.move footThe motion causes the foot to move from its initial position to its (cid:12)nal positionthrough an arc of a certain elevation. A height of 0 implies that the foot movesin straight-line path. If both the initial and (cid:12)nal positions are on the (cid:13)oor,then this means the foot will slide along the (cid:13)oor. A height of 10cm meansthe toes will reach a maximum height from the (cid:13)oor of 10cm halfway throughthe motion.The e(cid:11)ect of thecommand depends upon how the (cid:12)gurecreate foot motionis supported.Interactively, thecommand automatically sets themove footsupport of the (cid:12)gure to the moving foot, and thecommandcreate foot motiondoes the same. However, this doeshappen during the generation of thenotmovement sequence. The behavior of the feet depends very much on thesupport of the (cid:12)gure, although the e(cid:11)ect is quite subtle and di(cid:14)cult to de(cid:12)ne.A foot motion can move either the supported or non-supported foot, but it ismuch better at moving the non-supported one.The general rule of thumb for (cid:12)gure support during a movement sequenceis the opposite of that for interactive manipulation: during a movement se-quence, it is best to have the support through the foot on which the (cid:12)gurehas most of its weight. This will ensure that this foot remains (cid:12)rmly planted.The behavior of the feet can be changed at a speci(cid:12)c moment with thecreate timed foot controlcommand. This command needs starting and endingtimes and the type of control, just like thecommand. When theset foot controlmotion’s ending time is reached, the behavior reverts to its previous value.4.4.6 Moving the HeelsThe movement of the foot originates through the toes, but usually a steppingmotion begins with the heel coming o(cid:11) the (cid:13)oor. This may be speci(cid:12)ed withthecommand. This command does not ask for a location; itcreate heel motiononly asks for a height. A height of 0 means on the (cid:13)oor.Usually a stepping sequence involves several overlapping motions. It beginswith a heel motion to bring the heel o(cid:11) the (cid:13)oor, and at the same time a centerof mass motion to shift the weight to the other foot. Then a foot motion causesthe foot to move to a new location. When the foot is close to its new location,a second heel motion causes the heel to be planted on the (cid:13)oor and a secondcenter of mass motion shifts some of the weight back to this foot.132CHAPTER 4. BEHAVIORAL CONTROL4.4.7 The ArmsThe arms may be controlled through the command. Thiscreate arm motioncommand moves the arms to a point in space or to a reference point such as asite. The arm motion may involve only the joints of the arm or it may involvebending from the waist as well. The command requires the selection of theright or left arm and whether the arm movement is to be con(cid:12)ned to the armor include a bending of the torso. Arm movements involving the torso shouldnotcreate torso motionbe combined with a torso movement generated with thecommand. Both of these control the torso in con(cid:13)icting ways.The hand is then moved to the new position in space, using the sametechnique as thecommand. The user can specify if this position ismove armrelative to a segment; that is, to a global coordinate location or to a locationrelative to another ob ject.If the location is relative, the hand will moveto that ob ject even if the ob ject is moving as the hand moves during themovement generation.4.4.8 The HandsHand behavior may also be speci(cid:12)ed over time with thecreate timed handcontrolcommand. The hand can be temporarily attached to certain ob jectsover certain intervals of time. This command requires starting and endingtimes and the type of control, just like thecommand.set torso controlOb jects can be attachedover an interval of time with theto the handscreate timed attachmentcommand. The timing of the grasp action can be setaccordingly. During animation, one can specify the hand grasp site, the ap-proach direction, the starting hand pose, and the sequencing of (cid:12)nger motionsculminating in the proper grasp. If one is willing to wait a bit, the hand posewill even be compliant, via collision detection, to changes in the geometry ofthe grasped ob ject as it or the hand is moved.4.5 Virtual Human Control3We can track, in real-time, the position and posture of a human body, usinga minimal number of 6 DOF sensors to capture full body standing postures.We use four sensors to create a good approximation of a human operator’sposition and posture, and map it on to the articulated (cid:12)gure model. Suchreal motion inputs can be used for a variety of purposes.(cid:15)If motion data can be input fast enough, live performances can be ani-mated. Several other virtual human (cid:12)gures in an environment can reactand move in real-time to the motions of the operator-controlled human(cid:12)gure.3Michael Hollick, John Granieri4.5. VIRTUAL HUMAN CONTROL133Figure 4.11: Sensor Placement and Support Polygon.(cid:15)Motion can be recorded and played back for analysis in di(cid:11)erent environ-ments. The spatial locations and motions of various body parts can bemapped onto di(cid:11)erent-sized human (cid:12)gures; for example, a 5percentileththoperator’s motion can be mapped onto a 95percentile (cid:12)gure.(cid:15)Virtual inputs can be used for direct manipulation in an environment,using the human (cid:12)gure’s own body segments; for example, the handscan grasp and push ob jects.We use constraints and behavior functions to map operator body locationsfrom external sensor values into human postures.We are using thefrom Ascension Technology, Inc.toFlock of Birdstrack four points of interest on the operator. Sensors are a(cid:14)xed to the op-erator’s palms, waist, and base of neck by elastic straps fastened with velcro(Fig. 4.11). Each sensor outputs its 3D location and orientation in space.With an Extended Range Transmitter the operator can move about in an8-10 foot hemisphere. Each bird sensor is connected to a Silicon Graphics310VGX via a direct RS232 connection running at 38,400 baud.One of the initial problems with this system was slowdown of the simu-lation due to the sensors. The Silicon Graphics operating system introducesa substantial delay between when data arrives at a port and when it can beaccessed. This problem was solved by delegating control of the Flock to aseparate server process. This server will con(cid:12)gure the Flock to suit a client’s134CHAPTER 4. BEHAVIORAL CONTROLyzxyzxyxzfrontfrontfrontLateralAxialFlexionFigure 4.12: Extracting the Spine Target Vectorneeds, then provide the client with updates when requested. The server takesupdates from the Birds at the maximum possible rate, and responds to clientrequests by sending the most recent update from the appropriate Bird. Thisimplementation allows access to the Flock from any machine on the localnetwork and allows the client to run with minimal performance degradationdue to the overhead of managing the sensors. The sensors produce about 50updates per second, of which only about 8 to 10 are currently used due to thee(cid:11)ective frame rate with a shaded environment of about 2000 polygons. Thebulk of the computation lies in the inverse kinematics routines.The system must (cid:12)rst be calibrated to account for the operator’s size. Thiscan be done in two ways { the sensor data can be o(cid:11)set to match the model’ssize, or the model can be scaled to match the operator. Either approach maybe taken, depending on the requirements of the particular situation beingsimulated.Each frame of the simulation requires the following steps:1. The pelvis segment is moved as the (cid:12)rst step of the simulation. Theabsolute position/orientation of this segment is given by the waist sensorafter adding the appropriate o(cid:11)sets. The (cid:12)gure is rooted through thepelvis, so this sensor determines the overall location of the (cid:12)gure.2. The spine is now adjusted, using the location of the waist sensor andpelvis as its base. The spine initiator joint, resistor joint, and resistanceparameters are (cid:12)xed, and the spine target position is extracted from therelationship between the waist and neck sensors. The waist sensor givesthe absolute position of the pelvis and base of the spine, while the restof the upper torso is placed algorithmically by the model.The spine target position is a 3 vector that can be thought of as thesum of the three types of bending the spine undergoes { (cid:13)exion, axial,and lateral. Since the sensors approximate the position/orientation ofthe base and top of the spine, we can extract this information directly.Lateral bending is found from the di(cid:11)erence in orientation along thezaxis, axial twisting is found from the di(cid:11)erence inorientation, andy4.5. VIRTUAL HUMAN CONTROL135(cid:13)exion is determined from the di(cid:11)erence inorientation (Fig. 4.12).xNote that the \front" vectors in this (cid:12)gure indicate the front of thehuman. This information is composed into the spine target vector andsent directly to the model to simulate the approximate bending of theoperator’s spine.3. Now that the torso has been positioned, the arms can be set. Each armof the (cid:12)gure is controlled by a sensor placed on the operator’s palm.This sensor is used directly as the goal of a position and orientationconstraint. The end e(cid:11)ector of this constraint is a site on the palm thatmatches the placement of the sensor, and the joint chain involved is thewrist, elbow, and shoulder joint.4. The (cid:12)gure’s upper body is now completely postured (except for thehead), so the center of mass can be computed. The active steppingbehaviors are used to compute new foot locations that will balance the(cid:12)gure. Leg motions are then executed to place the feet in these newlocations.One unique aspect of this system is the absolute measurement of 3D carte-sian space coordinates and orientations of body points of interest, rather thanjoint angles. Thus, while the model’s posture may not precisely match theoperator’s, the end e(cid:11)ectors of the constraints are always correct. This is veryimportant in situations where the operator is controlling a human model ofdi(cid:11)erent size in a simulated environment.With a (cid:12)fth sensor placed on the forehead, gaze direction can be approxi-mated. Hand gestures could be sensed with readily available hand pose sensinggloves. These inputs would directly control nearly the full range ofbe-Jackhaviors. The result is a virtual human controlled by a minimally encumberedoperator.136CHAPTER 4. BEHAVIORAL CONTROLChapter 5Simulation with Societiesof Behaviors1Recent research in autonomous robot construction and in computer graphicsanimation has found that a control architecture with networks of functionalbehaviors is far more successful for accomplishing real-world tasks than tra-ditional methods. The high-level control and often the behaviors themselvesare motivated by the animal sciences, where the individual behaviors have thefollowing properties:(cid:15)they are grounded in perception.(cid:15)they normally participate in directing an agent’s e(cid:11)ectors.(cid:15)they may attempt to activate or deactivate one-another.(cid:15)each behavior by itself performs some task useful to the agent.In both robotics and animation there is a desire to control agents in en-vironments, though in graphics both are simulated, and in both cases themove to the animal sciences is out of discontent with traditional methods.Computer animation researchers are discontent with direct kinematic controland are increasingly willing to sacri(cid:12)ce complete control for realism. Roboticsresearchers are reacting against the traditional symbolic reasoning approachesto control such as automatic planning or expert systems. Symbolic reason-ing approaches are brittle and incapable of adapting to unexpected situations(both advantageous and disastrous). The approach taken is, more or less, totightly couple sensors and e(cid:11)ectors and to rely on what Brooks [Bro90] callsemergent behavior, where independent behaviors interact to achieve a morecomplicated behavior. From autonomous robot research this approach hasbeen proposed under a variety of names including:subsumption architectureby [Bro86],by [GL90, Kae90],by [AC87],reactive planningsituated activity1Welton Becket.137138CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSand others. Of particular interest to us, however, are those motivated ex-plicitly by animal behavior:by Brooks [Bro90],new AIemergent re(cid:13)exivebehaviorcomputational neuro-ethologyby Anderson and Donath [AD90], andby Beer, Chiel, and Sterling [BCS90]. The motivating observation behind allof these is that even very simple animals with far less computational powerthan a calculator can solve real world problems in path planning, motioncontrol, and survivalist goal attainment, whereas a mobile robot equippedwith sonar sensors, laser-range (cid:12)nders, and a radio-Ethernet connection to aProlog-based hierarchical planner on a supercomputer is helpless when facedwith the unexpected. The excitement surrounding the success of incorporat-ing animal-based control systems is almost revolutionary in tone and has ledsome proponents such as Brooks [Bro90] to claim that symbolic methods arefundamentally (cid:13)awed and should be dismantled.Our feeling, supported by Maes [Mae90], is that neural-level coupling ofsensors to e(cid:11)ectors partitioned into functional groupings is essential for thelowest(to use Brooks’ term), though by itself this purelylevels of competencere(cid:13)exive behavior will not be able to capture the long-term planning and pre-diction behavior exhibited by humans and mammals in general. Associationlearning through classical conditioning can be implemented, perhaps througha connectionist approach [BW90], though this leads only to passive statisticsgathering and no explicit prediction of future events.Our feeling is that symbolic reasoning is not (cid:13)awed, it is just not e(cid:14)-cient for controlling real-valued, imprecise tasks directly. The problem withtraditional planning is its insistence on constructing complete, detailed plansbefore executing. Recent research in this area has focused directly on relax-ing this constraint by interleaving planning and executing, reusing pieces ofplans, delaying planning until absolutely necessary, and dealing directly withuncertainty. The distinction between the symbol manipulation paradigm andthe emergent computation paradigm is even blurring|Maes has shown how atraditional means-ends-analysis planner can be embedded in an emergent com-putation framework, and Shastri [Sha88] has shown how simple symbol rep-resentation and manipulation can be accomplished in neural networks (whichcan be seen as the most (cid:12)ne-grained form of neuro-physiologically consistentemergent computation).Our strategy for agent construction will be to recognize that some formof symbolic reasoning is at the top motivational level and biologically-basedfeedback mechanisms are at the bottom e(cid:11)ector level. By putting them inthe same programming environment we hope to gain insight into how theseextremes connect. Hopefully, the result will be more robust than the harsh,rigid, feedback-devoid distinction between the planner and its directly im-plemented plan primitives. As will be discussed in Section 5.1.7, however, animportant technique for understanding what is missing will be to make prema-ture leaps from high-level plans to low-level behaviors appropriate for simplecreatures. This approach is bidirectional and opportunistic. Blind top-downdevelopment may never reach the real world and pure bottom-up developmentfaces the horror of an in(cid:12)nite search space with no search heuristic and no5.1. FORWARD SIMULATION WITH BEHAVIORS139clear goals.In this Chapter we (cid:12)rst pursue this notion of societies of behaviors thatcreate a forward (reactive) simulation of human activity. The remaining Sec-tions present some of the particular behaviors that appear to be crucial fornatural tasks, including locomotion along arbitrary planar paths, strengthguided motion, collision-free path planning, and qualitative posture planning.5.1 Forward Simulation with BehaviorsFigure 5.1 is a diagram of the control (cid:13)ow of a possible agent architecture.The cognitive model that will manage high-level reasoning is shown only as aclosed box. It will not be discussed in this section other than its input/outputrelation | it is the topic of Chapter 6. The importance of encapsulating thecognitive model is that it does not matter for the purposes of this section how itis implemented. Inevitably, there are direct links between the subcomponentsof the cognitive model and the rest of the system. However, we believe thelevel of detail of the current system allows ignoring these links without harm.The components of an agent are:1.: this will be discussed in Section 5.1.1, butSimulated Perceptionnote that raw perceptual data from the perception module is muchhigher level than raw data in a machine perception sense | our rawdata includes relative positions of ob jects and their abstract physicalproperties such as ob ject type and color. In a simulation we have per-fect environmental information, so it is the job of the sensors to alsosimulate realistically limited values.2.: perceptual behaviorsPerceptual (A(cid:11)erent) Behavior Networkthat attempt to to (cid:12)nd high-level information from raw sensory data.Typically they respond tosignals which change (cid:12)eld of view,focusingthresholds, distance sensitivity, restrictions on type of ob ject sensed,and the like.3.: the source of long-range planning and internal mo-Cognitive Modeltivation (activity not triggered directly by perception).4.: behaviors that derive activation or de-E(cid:11)erent Behavior Networkactivation signals. (Note that the a(cid:11)erent and e(cid:11)erent behavior net-works are separated only for organizational convenience | they couldactually be one network.)5.: attempt to modify ob jects embedded in theSimulated E(cid:11)ectorskinematics or dynamics simulation.Although there may be a general feed-forward nature through the abovecomponents in order, the connectivity must be a completely connected graphwith the following exceptions:140CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSSimulated
PerceptionPerceptual
(Afferent)
BehaviorsCognitive
ModelEfferent
BehaviorsSimulated
EffectorsFigure 5.1: Abstract Agent Architecture.SimulationCoreSimulationParticipantActionControllerKinematicObjectBehaviorsNetObjectsJackFigureDynamicObjectFigure 5.2: Outline of System Class Hierarchy.5.1. FORWARD SIMULATION WITH BEHAVIORS1411. The cognitive model cannot activate e(cid:11)ectors directly.2. There is no feedback directly from e(cid:11)ectors | e(cid:11)ector feedback is con-sidered perception (usually proprioception, though pain from musclefatigue is also possible) and is thus fed-back through the environment.Raw perceptual information may go directly to the cognitive model orto e(cid:11)erent behaviors, but it is typically routed through perceptual behaviorswhich derive higher level information and are sensitive to various focusingcontrol signals from the cognitive model, e(cid:11)erent behaviors, or perhaps evenother perceptual behaviors. The cognitive model may attempt to re-focusperceptual information through signals to the perceptual behaviors or it mayactivate or deactivate e(cid:11)erent behaviors in order to accomplish some type ofmotion or physical change. E(cid:11)erent behaviors may send signals to e(cid:11)ectors,send feedback signals to the cognitive model, or attempt to focus perceptualbehaviors.One typical pattern of activity associated with high-level motivation maybe that the cognitive model, for whatever reason, wants to accomplish a com-plex motion task such as going to the other side of a cluttered room containingseveral moving obstacles. The cognitive model activates a set of e(cid:11)erent be-haviors to various degrees, perhaps an ob ject attraction behavior (to get tothe goal) and a variety of obstacle avoidance behaviors. The e(cid:11)erent behaviorsthen continually activate e(cid:11)ectors based on activation levels from the cognitivemodel and from information directly from perceptual behaviors. Note thatthis (cid:12)nal control (cid:13)ow from perception directly to e(cid:11)erent behavior is whatis traditionally called. In another typical pattern of activity,feedback controlre(cid:13)ex behavior, e(cid:11)erent behavior is initiated directly by perceptual behaviors.Note, however, that especially in high-level creatures such as humans, the cog-nitive model may be able to stop the re(cid:13)ex arc through a variety of inhibitorysignals.5.1.1 The Simulation ModelRather than implementing models on real robots we will implement and testin detailed simulations that by analogy to the world have a physically-based,reactive environment where some ob jects in the environment are under thecontrol of agent models that attempt to move their host ob jects.For the agent modeler, the main advantage to testing in simulations is theability to abstract over perception. Because agents are embedded in a simu-lation, they can be supplied with the high-level results of perception directly,abstracting over the fact that general machine perception is not available. Atone extreme agents can be omniscient, having exact information about posi-tions, locations, and properties of all ob jects in the environment, and at theother extreme they can be supplied with a color bitmap image of what wouldappear on the agent’s visual plane. A good compromise that avoids excessiveprocessing but that also provides for realistically limited perception, is sug-gested by [Rey88] and also by [RMTT90]. They use the Z-bu(cid:11)ering hardware142CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSon graphics workstations (or a software emulation) to render a bitmap pro-jection of what the agent can see, except that the color of an ob ject in theenvironment is unique and serves to identify the ob ject in the image. Thecombination of the resulting image and the Z-bu(cid:11)er values indicate all visi-ble ob jects and their distances, and this can be used for ob ject location ordetermination of uncluttered areas.Many models of reactive agents are accompanied by a simulation with 2Dgraphical output such as [AC87, PR90, HC90, VB90], however, these sim-ulation environments are extreme abstractions over a real environment andassume discrete, two-dimensional, purely kinematic space. Such abstractionsare, of course, necessary in initial phases of understanding how to model anintelligent reactive agent, but extended use of a system without real-valuedinput parameters and immense environmental complexity is dangerous. Aswill be discussed Section 5.1.3, Simon [Sim81] argues that complex behavior isoften due to a complex environment, where the agent responds to environmen-tal complexity through simple feedback mechanisms grounded in sensation.When environmental complexity is not present, the agent modeler, noticingthe lack of complexity, may commit, also discussed in Sec-agent bloatingtion 5.1.3, where environmental complexity is accounted for arti(cid:12)cially in theagent model.5.1.2 The Physical Execution EnvironmentIn our model, kinematic and dynamic behavior has been factored out of theagent models and is handled by a separate, common mechanism. The networksof e(cid:11)erent behaviors controlling a conceptual agent in the environment willrequestmotion by activating various e(cid:11)ectors. The requested movement maynot happen due to the agent’s physical limitations, collision or contact withthe environment, or competition with other behavioral nets.Simulations of agents interacting with environments must execute on rea-sonably (cid:12)ne-grained physically-based simulations of the world in order to re-sult in realistic, useful animations without incurring what we call theagent-bloatingphenomenon, where motion qualities arising from execution in phys-ical environment are stu(cid:11)ed into the agent model. One of Simon’s centralissues [Sim81] is that complex behavior is often not the result of a complexcontrol mechanism, but of a simple feedback system interacting with a com-plex environment. Currently, for simplicity, our animations are done in akinematicenvironment (one considering only velocity and position) and notaone (also considering mass and force). Using only kinematics hasdynamicbeen out of necessity since general dynamics models have not been availableuntil recently, and even then are so slow as to preclude even near real timeexecution for all but the simplest of environments. Kinematic environmentsare often preferred by some since kinematic motion is substantially easier tocontrol with respect to position of ob jects since there is no mass to causemomentum, unexpected frictional forces to inhibit motion, and so on. But aswe demand more of our agent models we will want them to exhibit properties5.1. FORWARD SIMULATION WITH BEHAVIORS143that result from interaction with a complex physical world with endless, unex-pected intricacies and deviations from desired motion. Unless we execute on aphysically reactive environment we will experience one form of agent-bloatingwhere we build the physical environment into the agents. If we build an actualsimulation model into agents we have wasted space and introduced organiza-tional complexities by not beginning with a common physical environment.If we build the environmental complexity into the agents abstractly, perhapsthrough statistical models, we will have initial success in abstract situations,but never be able to drive a meaningful, correct, time-stepped simulation withmultiple agents interacting with an environment and each other. We do notmean that statistical and other abstract characterizations of behavior are notnecessary { just that abstract description is essential to understanding howthe underlying process works and judging when a model is adequate.The much cited loss of control in dynamic simulations needs to be over-come, and the message of emergent behavior research is that perhaps themost straightforward approach to this is by looking at the plethora of work-ing existence proofs: real animals. Even the simplest of single-celled crea-tures executes in an in(cid:12)nitely complex physical simulation, and creatures wenormally ascribe little or no intelligence to exhibit extremely e(cid:11)ective con-trol and goal-orientedness. Animals do this primarily through societies offeedback mechanisms where the lowest levels are direct sensation and musclecontraction (or hormone production or whatever).In our system dynamic simulations should enjoy the following properties:(cid:15)E(cid:11)ectors request movement by applying a force at a certain position toan ob ject.(cid:15)Collisions are detected by the system, which will communicate responseforces to those participating in the crash or contact situation.(cid:15)Viscous (cid:13)uid damping is simulated by applying a resistance force oppo-site and proportionate to instantaneous velocity.For simplicity, and especially when the motion is intended to be abstract,a simulation may still be run on a purely kinematic environment which hasthe following properties:cation of force.1. E(cid:11)ectors request changes in position and orientation, rather than appli-2. Every ob ject has some maximum velocity.3. No motion takes place unless requested explicitly by e(cid:11)ectors.4. Collisions are resolved by stopping motion along the system’s estimatedaxis of penetration.5. The system adapts the time increment based on instantaneous velocityand size of ob ject along that ob ject’s velocity vector so that no ob jectcould pass entirely through another ob ject in one time step.144CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSThe particular physical simulation approach is to use a simple (cid:12)nite-di(cid:11)erence approximation to the equations for elastic solids. Ob jects are mod-eled as meshes of point masses connected by springs (including cross con-nections to maintain shape), where tighter spring constants yield more rigidlooking bodies. This approach is discussed by Terzopoulos [TPBF87] andMiller [Mil91] and has the advantage of extreme simplicity and generality. Be-cause it is a discrete approximation to the \integral-level" analytical physicsequations it can solve many problems for free, though in general the cost islimited accuracy and much slower execution times than the correspondingdirect analytical methods (the results, however, are not only \good enoughfor animation" but are good enough considering our abstraction level). Themodel can easily account for phenomena such as collision response, elasticdeformation, permanent deformation, breakage, and melting. Finite elementanalysis yields a better dynamic behavior to the point-mass mesh (for accu-racy and execution time), but is not as general as the mass/spring approachand cannot model breakage and melting.5.1.3 Networks of Behaviors and EventsThe insulation of the cognitive model with networks of behaviors relies onemergent computation. It is important to understand, then, why emergentcomputation works where a strict hierarchy would not, and what problems anemergent computation approach poses for the agent designer and how theseproblems can be overcome.For simplicity, existing high-level task-simulation environments tend tomodel activity in strict tree-structured hierarchies, with competition occur-ring only for end e(cid:11)ectors in simulation models as in [Zel82], or for position ofa body component in purely kinematic models. However, for some time behav-ior scientists and those in(cid:13)uenced by them have argued that although there isobservable hierarchy, behavior { especially within hierarchical levels { is nottree structured but may have an arbitrary graph of in(cid:13)uence [Gal80, Alb81].In particular a theory of behavior organization must anticipate behaviors hav-ing more than one parent and cycles in the graph of in(cid:13)uence.The central observation is that in many situations small components com-municating in the correct way can gracefully solve a problem where a directalgorithm may be awkward and clumsy. Of course this approach of solvingproblems by having a massive number of components communicating in theright way is nothing new: cellular automata, fractals, approximation methods,neural networks (both real and arti(cid:12)cial), (cid:12)nite-di(cid:11)erence models of elasticsolids [TPBF87], simulated annealing, and so on use exactly this approach.The drawback to such massively parallel systems without central control istypically the inability to see beyond local minima. Certainly a high-level plan-ner may periodically exert in(cid:13)uence on various system components in orderto pull the system state from a local minimum. The appropriate introductionof randomness into component behavior, however, can help a system settlein a more globally optimal situation. This randomness can be from explicit5.1. FORWARD SIMULATION WITH BEHAVIORS145environmental complexity, introduction of stochastic components, limited orincorrect information, or mutation.This general approach is not limited to low-level interaction with the envi-ronment. Minsky proposes a model of high-level cognition in [Min86] where a\society of agents" interacts (organized as a graph) to accomplish high-levelbehavior. Pattie Maes [Mae90] has proposed an approach to high-level plan-ning through distributed interaction of plan-transformation rules. Ron Sunproposed a distributed, connectionist approach to non-monotonic reasoningAll of these approaches rest on emergent computation | behavior resultingfrom communication of independent components. Common ob jections to such1. it is doomed to limited situations through its tendency to get stuck in2. in order to implement, it requires an unreasonable amount of weight[Sun91].an approach are:local minima.(cid:12)ddling.The (cid:12)rst ob jection has already been addressed. The second is a seriousconcern. Our proposed solution will be to transfer the weight assignmentprocess to some combination of the behavioral system and its environment.An evolution model is one way to do this, as Braitenberg [Bra84] does withhis vehicles, or as the Arti(cid:12)cial Life (cid:12)eld would do. Another is to combinesimple behavioral psychology principles and a connectionist learning modelin a creature that wants to maximize expected utility [Bec92], then provide areinforcement model that punishes the creature whenever it does somethingwrong (like hits something).Making it easy for a human designer to engage in an iterative design andtest process is another approach. Wilhelms and Skinner’s [WS90] systemdoes exactly this by providing a sophisticated user interface and stressingreal-time or at least pseudo-real-time simulation of creatures interacting withthe environment. However, we will not pursue this approach for the followingreasons:(cid:15)Self-supervised weight assignment as agents interact with their environ-ment is clearly more desirable from a simulation point of view, thoughit sacri(cid:12)ces direct control for realism and ease of use.(cid:15)For reasons discussed in Section 5.1.2, we encourage execution in com-plex physically-based environments | an emphasis precluding real-timeplayback on standard displays.5.1.4 Interaction with Other ModelsOur approach then, is to control physical simulation with the abstract (cid:12)ndingsof the animal sciences, beginning by using the tricks that low-level animals146CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSuse. Low-level animal behavior tends, through its extensive use of environ-mental feedback, to be incremental | it makes a new decision at every mo-ment considering the current state of the environment. For this reason it isconsideredbecause it will incorporate unexpected events in constantreactivetime as though they had been planned for in advance. Certainly human be-havior exhibits short and long term planning that cannot be explained bypurely reactive processes. We hope to discover and elaborate abstraction lay-ers with long-term symbolic planning at the top and feedback mechanisms atthe bottom.However, there are countless areas in the neuro-physiological level study ofhumans that are not well enough understood to allow direct or even abstractimplementation. Behavior such as human walking, to our knowledge, can-not be described accurately in terms of feedback from proprioceptive sensorsand perhaps the vision system. Many such components of human behaviorcan, however, be modeled directly by abstract methods and we ought to beable to incorporate these as we progress. These direct methods will often beconsiderably faster than the corresponding neuro-physiological models whichtypically rely an massive parallel computation and will not run e(cid:14)ciently onsequential machines. So even if there were a neural-level walking algorithmfor humans, in cases where the robustness and correctness of locomotion areunlikely to contribute to the usefulness of the overall simulation, say becausethe are very few obstacles and the terrain is simple, it would be useful to usethe direct method to save time.Algorithms that are designed to manipulate human body parts directlycan be incorporated into the described system as long as an approach tocon(cid:13)ict resolution is also implemented should there be more than one behaviorattempting to control a given body segment (this can be weighted averaging,or strict prioritization, or whatever). Since the rest of the system is totallyreactive and considers the current state of the environment at every instant, itdoes not matter whether the physical model, kinematic model, or some otherprocess modi(cid:12)ed a given ob ject.As will be discussed later, if a collision is detected, all behaviors controllingthe o(cid:11)enders will be sent messages indicating the points of collision and theimpulse forces. For ob jects that do not have explicit velocity information avelocity is simply computed by looking at the system’s currentand how far(cid:14) tthe ob ject was moved over the previous time step. The receiving behaviorscan do whatever they wish with the information | replan, respond to it,ignore it, or anything appropriate. The only di(cid:14)culty is when two ob jectsboth controlled by unyielding direct control methods collide | they bothwill fail to move any further. This can be avoided by keeping the number ofob jects under direct control limited, or by always implementing some sort offailure recovery method. Since communication is based on contact positionsand forces, di(cid:11)erent control approaches can always communicate through theire(cid:11)ects on the environment.Even other physically-based ob ject controllers such as (cid:12)nite element anal-ysis, or direct rigid body dynamics can be incorporated. Direct control mes-5.1. FORWARD SIMULATION WITH BEHAVIORS147sages across computer networks or through operating system pipes to dynam-ics simulation packages can also be used. Direct manipulation is also possiblethough there must be a way to compensate for speed di(cid:11)erences if the simula-tion is running much slower than real time. One way to do this is to have theuser move an ob ject while the simulation is frozen, ask the user how long insimulation time that action should take, then use a direct kinematic controllerto do a spline-smoothed approximation of the user’s motion as the simulationcontinues.5.1.5 The SimulatorThe simulation of intelligent agents interacting with a reactive environmentis advanced incrementally in small adaptive time steps. The (cid:1)for a timetslice will be no greater thanth of a second (the typical video frame rate)and can be as small as (cid:13)oating point precision will allow. Typically, kine-130130matically controlled ob jects will update onths of a second but dynamicallycontrolled ob jects when experiencing high-impact collisions will want verysmall time steps. The distinction made in earlier sections between agents andthe environment is only conceptual at the simulator level| both componentsof agent models and components of physical models are considered (cid:12)rst-classparticipants in a single simulation. Every time step is broken down into anumber of synchronizing phases. The synchronizing phases are motivated byHaumann and Parent’s behavioral simulation system [HP88], but augmentedwith features for adaptive time steps. The following messages are broadcastin the given order to every participant on every time step:startThis tells participants a time step is beginning. Typically bu(cid:11)ersfor collecting messages in the a(cid:11)ect stage are cleared here, andstate information is saved in case there is a backup.a(cid:11)ectParticipants that attempt to modify the state of other partici-pants may do so here by looking at the state of the environmentand sending messages calling for change. However, no partic-ipant is allowed to change the appearance of an internal state{ all calls for change must be bu(cid:11)ered and dealt with in therespond stage.respondOb jects are allowed to change their externally accessible statevariables, such as position and color for environmental ob jectsor activation level for behavioral network components.data injectRigid rules, such as static non-interpenetration, are enforcedhere after ob jects have had a chance to update themselves. Purekinematic scripting may be done here also.In addition, at any phase any ob ject may request that the time step berestarted with a smaller time step if it feels the simulation is running too fast.A participant need only suggest a new (cid:1), perhaps half the previous (cid:1), thentt148CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORScall for a global backup. All participants are required to store persistent stateinformation in the start phase, and restore this state if a backup is called.Participants may also request new (cid:1)values without requesting a backuptand if no requests are made, the system will try to double the (cid:1)on everytstep until it reachesth of a second.130The Class HierarchyAn ob ject-oriented approach is natural for implementing such a system, andin order to allow fast execution for interactive development, we chose to useC++. The cognitive model is based on Lisp and Prolog and will communicatethrough C-callout functions to C++.The class hierarchy is outlined in Figure 5.2. TheclassSimulationCoremanages the clock, the current (cid:1), and a list of participants to which it broad-tcasts synchronizing messages on each time step.en-SimulationParticipantcapsulates all participants in the simulation. A distinction is made betweenparticipants that have spatial qualities (theclass, which tendsKinematicObjto operate in the respond stage) and participants that try to modify the stateof other participants (theclass which operates in both theActionControllera(cid:11)ect and respond stages).Ob jects, actions, and networksTheclass is broken into aclass which allows useKinematicObjJackFigureof the(cid:12)gures. Theclass is for ob jects controlled by dynamicDynamicObjJacksimulation and is a subset of kinematic ob jects because any dynamic ob jectought to be able to respond to any message intended for a purely kinematicob ject.ActionControllersare broken down intowhich include per-Behaviorsceptual and e(cid:11)erent behaviors discussed above andwhich connectNetObjectsthem. Our network package allows general neural network-type constructions,though it is important to note that the system is not a neural network because:(cid:15)the ’neurons’ (nodes) may be of arbitrary complexity.(cid:15)the messages passed along network links may be very complicated, inparticular, they can be pointers to ob jects.Neural networks can be used within behaviors, however, and we have be-gun experimenting with backpropagation learning [FS91, HKP91] and recur-rent networks [Bec92, Sch90] as ways of learning how to behave.Allinstances must respond to messages requesting theActionControllerstart and end time of activation or an indication that the action has not startedor ended. This allows general implementation of conditional action sequencingthrough meta-behaviors that on each time step check to see of a particularaction has started or ended or if an action of a particular type has begun orended.5.1. FORWARD SIMULATION WITH BEHAVIORS1495.1.6 Implemented BehaviorsPresently, reactive agents resemble Reynolds’ birds [Rey87] in that on eachtime step the agent moves along its local-axis. E(cid:11)erent behaviors attemptzto modify the global orientation of the local-axis and also determine by howzmuch it will move forward.Our primary perceptual behavior is thesensor. Its argumentsClosest-kare the number and type of ob jects to which it is sensitive. In addition thesensor needs to know what its relative position is to its host environmentalob ject (the-axis of this transformation will be the forward direction). Thezsensor producesgroups of outputs which will contain information on thekclosestob jects of the de(cid:12)ned type. Each group will have three (cid:13)oating-pointkoutput nodes: the distance from the sensor’s current global origin, the anglebetween the sensor’s-axis and a vector to the detected ob ject’s centroid, andzthe radius of the detected ob ject (we currently use bounding cylinders andbounding spheres around ob jects to simplify calculations). We have found nopressing need yet to construct a correspondingsensor.furthest-kAnother perceptual behavior that is not behaviorally motivated, but usefulfor abstract control is thethat is sensitive only to a particularObjectSensorob ject in the environment and has outputs similar to asensor.closest-kThe e(cid:11)erent behaviors we have implemented are loosely motivated byneuro-ethological (cid:12)ndings about real creatures and are discussed by Wilhelmsand Skinner [WS90] in their implementation and abstraction of Braitenberg’sVehicles[Bra84] and in Anderson and Donath’s emergent re(cid:13)exive behaviorsystem [AD90]. What is novel about our behaviors is their dynamically ad-justable tuning parameters:1. threshold distance2. (cid:12)eld of view (angular threshold)3. sensitivity of activation level to distance of ob ject and angular distanceof ob ject. Distance from threshold and angle from center scaled by anexponent and a constant (both focusing parameters).these can be adjusted directly by the cognitive component or by other behav-iors. We have found the following behaviors particularly useful:Attractgo toward either closest-k of a certain type of ob ject or a speci(cid:12)cob ject. Loosely corresponds toin animals [Gal80].teleotaxisAvoidgo away from a particular ob ject or from a certain type of ob ject.Also a form of teleotaxis.GoArounduses the incremental obstacle avoidance approach outlined byReynolds [Rey88], which is based on how birds avoid obstacleswhile (cid:13)ying.150CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSAchievego to a particular place or a particular place relative to anotherob ject.AttractLinego directly towards a line in space { used to follow walls.AvoidLinego directly away from a line in space { used to avoid walls.5.1.7 Simple human motion controlAn important approach to developing human behaviors is to attempt to applythe behaviors appropriate for low level animals directly to humans and seewhere they appear awkward in order to understand what is missing. Ourinitial attempt allows the above e(cid:11)erent and perceptual behaviors to be useddirectly in a human model except that instead of simply moving along the-zaxis the human agent attempts to reduce its current and desired headings bytaking steps. The stepping is accomplished by using a locomotion algorithm(Section 5.2). Our walking algorithm is incremental in that it only needs toknow where the next footstep should go and how it should be oriented. Ourapproach to supplying footsteps clips the di(cid:11)erence between the current anddesired headings to 45 degrees and places the next foot to be moved alongsidethe new heading. Heading is determined by the orientation of the lower torso(which is oriented by the walking algorithm). The size of the step is currentlybased on the curvature of the turn (smaller steps for larger turns), thoughcertainly step length should have other in(cid:13)uences.Simulations involving this simple human agent model show very clearlythat humans anticipate the e(cid:11)ects of placing the next step (perhaps throughbehavioral gradient estimation) rather than blindly following in(cid:13)uences of thecurrent situation. In all complicated situations under the described model theagent tends to oscillate around the behavior gradient.There are other agent behaviors that are in the process of migratinginto this behavioral framework. In the next sections we look at locomotion,strength-guided lifting, collision avoidance, and posture planning.5.2 Locomotion2Locomotion provides a tremendous extension of the workspace by movingthe body to places where other activities may be accomplished. A locomotionsystem should provide a reasonable con(cid:12)guration of the (cid:12)gure at any time asit moves along a speci(cid:12)ed input path. There have been many e(cid:11)orts to makethis process more realistic and automatic, which can roughly be summarizedinto two ma jor approaches: kinematic and dynamic controls. Rotoscopy dataand biomechanics knowledge can be utilized to control locomotion kinemati-cally, but empirical data must be generalized to get walking under parametriccontrol. A dynamics computation can be done to get the locomotion pathand some of the body motion characteristics, but biomechanics knowledge is2Hyeongseok Ko.5.2. LOCOMOTION151HSTOHSLEFT LEGRIGHT LEGTOHSDSDSINTERVAL1INTERVAL2HS=HEELSTRIKE
TO=TOE OFF
DS=DOUBLE STANCEFigure 5.3: The Phase Diagram of a Human Walk.useful in determining the details and reducing the complexity of the wholebody dynamic system.These two approaches can be applied to get straight path walking. Thenatural clutter and constraints of a workplace or other environment tend torestrict the usefulness of a straight path so we must generalize walking tocurved paths. We have already seen the stepping behavior and the collisionavoidance path planning in, so a locomotion capability rounds out theJackability of an agent to go anywhere accessible. First we give some necessaryde(cid:12)nitions for the locomotion problem, then look at feasible ways of imple-menting curved path walking.At a certain moment, if a leg is between its own heelstrike (beginning)and the other leg’s heelstrike (ending), it is called the. If a leg isstance legbetween the other leg’s heelstrike (beginning) and its own heelstrike (ending),it is called the. For example, in Figure 5.3, the left leg is the stanceswing legleg during interval 1, and the right leg is the stance leg during interval 2. Thusat any moment we can refer to a speci(cid:12)c leg as either the stance or swing legwith no ambiguity. The joints and segments in a leg will be referenced withpre(cid:12)xesor: for example, swing ankle is the ankle in the swingswingstanceleg.Let (cid:2) = [] be the joint angles and (cid:3) = [] be the links of(cid:18)1 ; : : : ; (cid:18)l1 ; : : : ; lJSthe human body model. Eachcan be a scalar or a vector depending on thei(cid:18)DOFs of the joint. Let (cid:6) be the sequence of ()= 01,h;d; sf; lorr; i;; : : : ; n~~iiiiwhereis the heel position of theth foot,is the direction of theth foot,iihiidsfiiis the step frequency of theth step, and(\left" or \right") is 0 whenilorrtheth foot is left foot and 1 otherwise. The locomotion problem is to (cid:12)ndithe functionthat relates (cid:3) and (cid:6) with (cid:2) at each timeft:(cid:2) =((cid:3)(cid:6))f;; t:(51):Usually the functionis not simple, so the trick is to try to devise a setfof algorithms that computes the value of (cid:2) for the given value of ((cid:3)(cid:6)),;; tdepending on the situation.5.2.1 Kinematic ControlThe value of (cid:2) can be given based on rotoscopy data. Two signi(cid:12)cant prob-lems in this approach are the various error sources in the measurements and152CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSthe discrepancy between the sub ject’s body and the computer model. Whenapplying kinematic (empirical) data to the model, obvious constraints im-posed on the walking motion may be violated. The most fundamental onesare that the supporting foot should not go through nor o(cid:11) the ground in theobvious ways depending on the situation, and that the global motion shouldbe continuous (especially at the heel strike point). The violation of these con-straints is visually too serious to be neglected. During motion generalizationthe error is likely to increase. Therefore in the kinematic control of locomo-tion, one prominent problem is how to resolve errors and enforce constraintswithout throwing away useful information that has already been obtained.So how can we generalize empirical data? The walk function (cid:2) depends onmany parameters, and simple interpolation cannot solve the problem. Boulic,Magnenat-Thalmann and Thalmann’s solution for this problem [BMTT90] isbased on the relative velocity (), which is simply the velocity expressed inRVterms of the height of the hip joint(e.g. 2). For example the heightHH=secttof the waistduring the walk is given byOs(cid:0):RV:RV(cid:25)t (cid:0):0015+ 0015sin 2(2035)whereis the elapsed time normalized by the cycle time. Because this formu-tlation is based on both body size and velocity, the approach can be appliedunder various body conditions and velocities.5.2.2 Dynamic ControlBruderlin and Calvert built a non-interpolating system to simulate humanlocomotion [Bru88, BC89]. They generated every frame based on a hybriddynamics and kinematics computation. They could generate a wide gamutof walking styles by changing the three primary parameters step length, stepfrequency, and speed. The example we use here is based on their work.Their model is divided into two submodels. The one (stance model) con-sists of the upperbody and the stance leg. The other (swing model) representsthe swing leg. In the stance model, the whole upperbody is represented byone link and the stance leg is represented with two collinear links joined by aprismatic joint. So the stance leg is regarded as one link with variable length!. In the swing model, the two links represent the thigh and calf of the swingleg. In both models, links below the ankle are not included in the dynamicanalysis and are handled instead by kinematics.Two sets of Lagrangian equations are formulated, one set for each legphase model. To do that, the (cid:12)ve generalized coordinates,,,,,!(cid:18)1(cid:18)2(cid:18)3(cid:18)4are introduced:is the length of stance leg;,,is measured from the!(cid:18)1(cid:18)2(cid:18)3vertical line at the hip to the stance leg, upperbody, and the thigh of theswing leg, respectively; andis the (cid:13)exion angle of the knee of the swing(cid:18)4leg. During the stance phase the stance foot remains at (), soandx; yxyare regarded as constants. Once those (cid:12)ve values of general coordinates aregiven, the con(cid:12)guration of the whole body can be determined by kinematics.5.2. LOCOMOTION153So the goal of the dynamics computation is to obtain the general coordinatevalues.!(cid:18)(cid:18)FF1F2,, and.We will focus only on the stance model here. By formulating the La-grangian equation on the stance model, we get the three generalized forcesFm2! (cid:0) m2 r2(cid:18)2(cid:18)2 (cid:0) (cid:18)1(cid:0) m2 r2(cid:18)2(cid:18)2 (cid:0)(cid:18)1(cid:18)2 (cid:0) (cid:18)1=(cid:127)sin()() cos()!(cid:127)___(cid:0)m2!(cid:18)1(cid:0) m2 r2(cid:18)2(cid:18)1(cid:18)2 (cid:0) (cid:18)1cos()___+cosm2g(cid:18)1(5.2)(cid:18)F1= (++)+ 2_I1m1 rm2!(cid:18)1m2!!(cid:18)11(cid:127)_22(cid:0)m1 r1m2!g(cid:18)1m2 r2(cid:18)2 !(cid:18)2 (cid:0) (cid:18)1(+)sin+cos()22_(cid:0)m2 r2(cid:18)2!(cid:18)2 (cid:0) (cid:18)1sin()(5.3)F2(cid:0)m2 r2(cid:18)2 (cid:0) (cid:18)1m2 r2!(cid:18)1(cid:18)2 (cid:0) (cid:18)1=!
(cid:127)sin() +cos()(cid:18)+(+)sinI2m2 r(cid:18)2 (cid:0) m2gr2(cid:18)222(cid:127)(cid:127)(cid:127)2+2_cos() +sin()(5.4)m2 r2!(cid:18)1(cid:18)2 (cid:0) (cid:18)1m2 r2!(cid:18)1(cid:18)2 (cid:0) (cid:18)1__which can be written in a matrix form asa11 a12 a13Fb1+!(cid:127)w232323a21 a22 a23(cid:18)1=F1b2+(55):(cid:127)(cid:18)a31 a32 a33(cid:127)(cid:18)2F2b3+(cid:18)454545Let() be the value ofat time;can be a scalar, a vector, or a matrix.xtxtxIn equation 5.5 at time, everything is known except (cid:127). Ift!;(cid:18)1 ;(cid:18)2; F; F1 ; F2(cid:127)(cid:127)w(cid:18)(cid:18)we give the generalized force values at time, the above linear equation cantbe solved for accelerations. The position at the next time step is~r~~rrqttqttqt_(+ (cid:1)) =_() + (cid:1)(cid:127)()(5.6)~qtt~qttqt:(+ (cid:1)) =() + (cid:1)_()(5.7)~rTThe internal joint torques() = []are not fully known.FtF; F1 ; F2w(cid:18)(cid:18)Bruderlin adopted some heuristics from biomechanics to handle this problem.For example, to get, he notedF1(cid:18)A signi(cid:12)cant torque at the hip of the stance leg occurs only justafter heel strike and lasts for about 20% of the cycle time. Also,the torque during this time interval is such that it rapidly reachesa maximum value and decays quickly towards the end [IRT81,Win90].and approximated it by a constant function(cid:18)F1=(58):cfor the (cid:12)rst 20% of cycle time(cid:26)0for the remaining cycle time.154CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSSimilarly, in modeling the hip joint and waist joint as springs, the internaltorques were given by the following formulas:F=(k!+)pa3 (cid:0) !(cid:0) v!
_(59):!!des!(cid:18)des_F2(cid:0)k2(cid:18)2 (cid:0) (cid:18)2(cid:0) v2(cid:18)2 :=()(510):To handle the errors coming from these approximations, several check-points were set. For example, the posture at the heel strike after the currentstep can be derived based on the step symmetry. Integrating() until theFtcheckpoint, we can compare the result with the desired one. The constants inthe equations above (e.g.) are adjusted according to the di(cid:11)erence. Thisc; a3process is repeated until the integration brings it close enough to the desiredposture.5.2.3 Curved Path WalkingResearch on biped locomotion has focused on sagittal plane walking in whichthe stepping path is a straight line. Unfortunately, simply treating a complexwalking path as a sequence of straight-line path segments does not work.The problems of turning and coordinating the limb motions at the turns isfrequently neglected and the rigid appearance of the resulting abrupt mid-airturns is clearly unacceptable animation.In building a general planar locomotion behavior, we utilized pre-existingstraight path ideas. We will call a linear path locomotion algorithm a 1Dsystem; we will use it as a process within our 2D behavior. For every 2D step,we will consider its, and the 1D system will provide someunderlying 1D stepneeded information. Our generalization algorithm from 1D to 2D is basedon the intuition that there should be a smooth transition between linear andcurved locomotion. If the curvature is not large, the 2D walk generated shouldbe close to the 1D walk given by the underlying 1D system. In particular,the degenerate 2D case of a straight line should be exactly the same as thatproduced by the underlying 1D system. Since no assumptions are made aboutthe underlying 1D system, any 1D locomotion algorithm can be generalizedinto our 2D one. Moreover, the underlying 1D system will determine thestylistics (or faults) of the curved path walk.When requested to create a step, the 1D step generation algorithm providesinformation to the 2D system. The 2D system (cid:12)rst computes the center [ofmass] site tra jectory and the locations of both hip joints. The locations ofthe feet are computed based on the 1D step information. Because we havethe hip and foot locations of both legs, the con(cid:12)gurations of both stance andswing legs can be determined. The banking angle is computed, and the upperbody is adjusted to move the center of mass to achieve the required banking.The parameters (cid:2) that determine the con(cid:12)guration of the whole body is nowavailable fordisplay. This entire process is incremental at the step level,Jackso that it (cid:12)ts neatly into the behavioral simulation paradigm.5.2. LOCOMOTION155di-1hi-1EiihdiHSMEi-1TOM12Figure 5.4: The Step Length of the Underlying 1D Step.Specifying the WalkThe direct input to the locomotion behavior is a(cid:6) of 4-tuples,step sequenceiiiii~~(cid:27)h;d; sf; lorr; i; : : : ; n:= ()= 0(511):Each tupleis called theth. The pair of adjacent two footi(cid:27)ifoot descriptiondescriptions () is called theth(cid:27)1 ; (cid:27)istep descriptionor simply theth.istepi(cid:0)iEven though we have maximum control of locomotion by using the stepsequence, generating such a sequence directly is a tedious job. The behavioralsimulation can generate a path incrementally, or an interactive user couldspecify a curved path. In either case, the speci(cid:12)cation is automatically trans-formed to a step sequence.The speci(cid:12)cation of a walk in 1D can be done by giving a sequence of()= 1. Each () a(cid:11)ects the type of current step, start-sl; sf; i; : : : ; nsl; sfiiiiing from the current heelstrike to the next one. For every step description() in 2D, we consider its. The step frequencyunderlying 1D stepsf1Di(cid:0)i(cid:27)1; (cid:27)of this 1D step is given byof. We can draw 2 lines,on thesf(cid:27)(cid:11)1(cid:11)2iihorizontal plane as shown in the Figure 5.4:is in the direction of(cid:11)1displaced byfrom;is in the direction ofdisplaced byfrom.(cid:21)h1(cid:11)2(cid:21)h~i(cid:0)~id~i(cid:0)d1~iLetbe the arc length of the spline curve fromto, whereand(cid:14)E1EE1Ei(cid:0)ii(cid:0)iare the pro jections of the heel positions to the lines, and, respectively.(cid:11)1(cid:11)2(The derivatives at the end points of this spline curve are given byand~i(cid:0)d1~id.) The step lengthof the underlying 1D step is given by this.(cid:14)sl1DPath of the Center SiteFor this discussion we will keep the model simple by assuming that the centerof mass moves along a straight line from the heelstrike moment of the stanceleg (HS) to the toe o(cid:11) moment of the swing leg (TO) (Figure 5.5). Thetra jectory of the center site during this double stance phase (DS) is givenby the current stance foot direction. From the TO to the next HS, the~i(cid:0)d1center site moves along a spline interpolation (Figure 5.5). At both ends of thespline, the derivative of the must match that of the adjacent line segments forllaad156CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSHSMTOMTOMHSMHSMTOMFigure 5.5: The Tra jectory of the Center Site (Top View).dhi-1i-1HSMC NHSMYhidiX1C TOMTOM2Figure 5.6: The Position of Center Site at Heelstrike Moment.(cid:12)rst order continuity. Through the whole locomotion, the pelvis is assumedto face the derivative direction of the center site path, and be vertical to theground. The torso can be bent in any direction, a part of which is givenby the underlying 1D algorithm, and another part is given from a bankingadjustment.To derive the spline curve, we need the positionand derivative_~CN HST OT Onextof the center site at theHS, as well asandat TO whichCC~CN HS~~_are provided by the underlying 1D system (Figure 5.6). The assumptionsabove imply thatC=, anddCshould be put at pointsomewhereX_~N HSiN HS~~on the line. Letandbe the length of the center site tra jectory from(cid:11)2(cid:17)1(cid:28)1DDHS to TO, and from TO to the next HS, respectively, during the underlying1D step. Letof corresponding 2D step be similarly de(cid:12)ned. Let(cid:17)2D(cid:28)2()XDbe the arc length (top view) of the spline fromtoin Figure 5.6. NowCX~T Othe position of the center siteat the next HS is set to the pointonX~CN HSthe linesuch that(cid:11)2(cid:17)1(cid:17)2DD(cid:28)1(cid:28)2()XDD=:(512):This de(cid:12)nition ofis based on the smooth transition assumption from~CN HS1D locomotion to 2D. By a mapping which preserves arc length ratio [Gir87,SB85, Far88], we can (cid:12)nd the correspondence between the 2D tra jectory ofthe center site and underlying 1D one. Note that this de(cid:12)nition also makesllaa5.2. LOCOMOTION157HSMTOMFigure 5.7: Banking of the Center Site.swHHSM2DCBANKINGHswTURNINGHSM2DCHSMTOMFigure 5.8: Turning of the Center Site.the degenerate case of 2D walk exactly same with the corresponding 1D walk.The displacement of the curved path from the underlying linear path isproduced by banking as shown in Figure 5.7. The position of the displacedcenter site in 2D step is, andis the position of the swing hip. BankingC2HDswmostly results from ankle joint adjustment. Even though the center site is puton the spline curve by the ankle angle, the upper body has not bent yet togenerate the overall correct banking of the whole body. The banking shouldbe considered in terms of the center of mass of the body. The overall bankingis given by(cid:30)= arctan()(513):2(cid:20)vgwhereis the velocity,is the gravity, andis the curvature of the pathvg(cid:20)[Gir87]. Here we use the spline curve of the center site as an approximation toget the curvature. The upper body should be bent so that the center of mass(which is in the upper body) may make the anglearound the stance ankle(cid:30)with respect to the ground.Iteration can be used to compute the currentcenter of mass and reduce the di(cid:11)erence from the current one and the desiredone.We assume that the length of the dynamic legin 2D locomotion at a!moment, is the same asat the corresponding moment in the underlying 1Dt!step. So the displaced center sitewill be lower than the corresponding 1DC2Dcenter site. The position () of the hypothetical ankle is availablex1 ; y1; z1C1Dfrom the old foot location. Let () be the position of the swing hipx2; y2 ; z2Hswin Figure 5.7. The horizontal componentsandcan be computedx2z2l158CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSfrom the derivative atand. In Figure 5.7,is the distance betweenC2(cid:21)(cid:21)DHC2swDand, and this distance is along the perpendicular direction of thederivative.Because we assumed thatis the same in 1D and 2D locomotion, we have!jx1; y1 ; z1(cid:0)x2; y2 ; z2j!()()=(514):whereis given by the underlying 1D system. The value ofthat satis(cid:12)es!y2the above equation is the height of bothand. Because the pelvis isHC2swDassumed to be upright through the steps, the height of the stance hipisHalso.y2The Stance LegThe center site movement of 2D locomotion during DS is the same as that ofthe 1D one, including the height component, so the stance leg con(cid:12)gurationsare given by the underlying 1D system. During the single stance phase, westill use 1D system to get the joint angle at the ball of foot. But because thecenter site begins to deviate, the other joint angles should be computed.In the stance leg, after the foot is put (cid:13)at on the ground, the toetip isregarded as the root because that point is not moved until the next toe o(cid:11).Because the joint angle at the ball of the foot is provided by the 1D algorithm,we have the locationof the ankle. Since the location of the hip is alsoAavailable the con(cid:12)guration of the stance leg can be determined.The Swing Leg at the Double Stance PhaseBecause a revolute joint is assumed at the ball of foot, if we exclude thepossibility of sliding, the toe group of the swing foot should stay (cid:12)xed on theground during the DS. Because there are 3 links between the swing hip andthe ball of foot, we should resolve the redundancy in a reasonable way. If weuse the ball of the foot joint angle in the 1D algorithm this redundancy goesaway. This approximation works well in most of the cases. But when boththe direction change and the step length (the distance between the adjacentsteps) are extremely large, the distancefrombecomes tooj ~(cid:26)jHtoAswswswlong to be connected by the lengths of thigh and calf. This problem is solvedby increasing the angle at the ball of foot untilbecomes less than thej ~(cid:26)jswsum of thigh and calf. Thenis used to get the joint angles at ankle, knee,sw~(cid:26)and hip.The Swing Leg at the Single Stance PhaseThe tra jectory (top view) followed by the swing ankle is approximated by asecond degree Casteljau curve [Far88]. The 3 control points are given by thepositionof the current swing ankle at TO,which is the symmetricD1D2point of the stance ankle with respect to the line, and the ankle position(cid:11)D3at the next heel strike point (Figure 5.9).5.2. LOCOMOTION159SWING FOOT (AFTER)D
3D
2STANCE FOOTSWING FOOT (BEFORE)D
1Figure 5.9: The Path of the Swing Foot.The height component of the ankle is determined by the underlying 1Dalgorithm. So now we have the locationsand, and can determine theAHswswswing leg con(cid:12)guration except for the two indeterminacies. At the moment ofheelstrike, the swing leg should have been prepared for the next step. Becauselinear walking is assumed from HS and TO, the hip and ankle angles in swingleg should become 0 except for the bending direction. So we should somehowadjust the swing leg and foot from the(at TO) to thechaos con(cid:12)gurationordered con(cid:12)guration(at the next HS).At toe o(cid:11), the displaced (rotated around the forward axis) foot gets tothe normal position very quickly and it is approximated by an exponentialfunction that decreases rapidly to 0: for some positive constant, we letG(cid:18)x3at timebet(cid:0)Gttsw(cid:22)(cid:22)(cid:18)t(cid:18)(cid:1)() =(0)exp(515):whereis the elapsed time after TO, andis the duration between the TOtswtand the next HS. The rotation of the swing leg around the the axis from theswing hip to the swing ankle is approximated by a parabola given by~~swt(cid:0) t2swt(cid:18)t(cid:18)(cid:1)() =(0)()(516):If the 2D locomotion path is actually a straight line, the walk generatedis exactly the same as the walk given by the underlying 1D system. Forexample, in the consideration of swing leg motion during the single stancephase,, andwill be collinear and parallel to the walking path.D1 ; D2D3xzAlso in the equations 5.15 and 5.16, both(0) and(0) will be 0 and the(cid:18)3(cid:18)7tra jectory of the swing leg will be exactly the same as that of 1D walking.5.2.4 ExamplesFigure 5.10 shows foot steps generated by the interactive. Fig-step editorure 5.11 shows the walking path generated by the interactive. Fig-path editorlla160CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.10: Steps Generated by the Step Editor.Figure 5.11: Steps Generated by the Path Editor.5.3. STRENGTH GUIDED MOTION161ure 5.12 shows snapshots during a turning step. Finally, Figure 5.13 shows apath generated incrementally by the approach and avoidance behaviors.5.3 Strength Guided Motion3Human motion is likely to be hybrids of many motion and path generationtechniques. The task is to (cid:12)nd e(cid:11)ective combinations that provide realis-tic motion behaviors while simultaneously o(cid:11)ering the user reasonable andintuitive control mechanisms. We have already seen several methods usingconstraints and other simple control methods that implement variousbasichuman activities such as reaching, looking, grasping, and balancing. Now wewill look closer at a task-level control algorithm for ob ject manipulation byan end-e(cid:11)ector, such as lifting a load to a speci(cid:12)ed position in space. The re-sulting motion is certainly dictated by the geometric limits and link structureof the body; but more importantly the motion is strongly in(cid:13)uenced by thestrengthcomfortandof the agent.5.3.1 Motion from Dynamics SimulationTorques may be used to physically simulate motions of a (cid:12)gure. Typicallythe joint responses are conditioned by springs and dampers so that responsesto external forces can be computed. Such force- and torque-based methodsare called dynamic simulations [Gir87, Gir91, AGL87, IC87, WB85, Wil87,FW88, Wil91, Hah88, HH87, Bar89]. Solving the dynamic equations, an initialvalue problem, is computationally expensive, especially if the joints are sti(cid:11)[AGL87]. Natural external forces such as gravity and collision reactions easilyyield motions which are free-swinging or passive (purely reactive), but whichgive the unfortunate impression of driving a hapless mannequin or puppet.As might be expected, the best examples of dynamic simulation come fromcrash studies [Pra84] where rapid deceleration produces forces that typicallyoverwhelm any agent-initiated torques. In less violent motions, the torquesmay be derived from a spring or vibration model. Such have been used tocreate convincing motions of worms, snakes, and other (cid:13)exible ob jects [Mil88,PW89], but this cannot be the same mechanism used for human (cid:12)gure motion.Dynamic simulations are annoyingly di(cid:14)cult to control by an animator sinceforce space speci(cid:12)cations are highly non-intuitive.Kinematic and inverse kinematic approaches are easier to manipulate andmay create the right \look," but su(cid:11)er from potentially unrealistic (or un-speci(cid:12)ed) velocities or torques in the body joints. These problems have beenaddressed as boundary value problems with ob jective functions. The tra jec-tories are then solved by global optimization approaches [WK88, Bre89] orcontrol theory [BN88], but their methods presume complete knowledge of thedriving conditions and overall constraints.3Philip Lee.162CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.12: Snapshots during a Turning Step.Figure 5.13: Incremental Walk Behavior.5.3. STRENGTH GUIDED MOTION163Robotics emphasizes accomplishing a motion within constraints and op-timizing it with respect to some criteria such as time, torque, energy, orobstacles [Bob88, HS85b, KN87, Kha87, MK85, SL87]. Bioengineers try todetermine if human motion conforms to some optimality criterion, such astime or energy [BC68, CJ71, YN87, Yeo76]. Given its range and diversity,human motion is not optimal with respect to a single criteria.Despite these various approaches to the human motion problem, none hasbeen successful at specifying a task by describing a load and a placementgoal, and then completing the task in a realistic (though possibly suboptimal)manner. There have been e(cid:11)orts to generate a path between two endpoints[AAW74, Ayo91, SH86, KR79, SSSN85], but the usual solution incorporatesconstraints and a single ob jective function that is optimized.5.3.2 Incorporating Strength and Comfort into MotionWe o(cid:11)er a solution which blends kinematic, dynamic and biomechanical in-formation when planning and executing a path. The task is described by thestarting position, the load (weight) that needs to be transported, and a goalposition for the load. Some simple additional parameters help select fromthe wide range of possible paths by invoking biomechanical and performanceconstraints in a natural fashion. Thus a path is determined from a generalmodel rather than provided by default or by an animator. In addition, thealgorithm is incremental: it has the ability to adapt to changing forces thatare required to complete a task. The basic premise of the method is that aperson tends to operate within aregion which is de(cid:12)ned bycomfortavailablestrength. This is even more probable when the person has to move a heavyob ject.We assume that a person tends to operate within a comfort region dic-tated by muscular strength, especially when moving a heavy ob ject. When aperson has to accomplish a given task, say lifting a box or cup, he starts fromsome initial posture and then plans the direction for his hand to move. Thisplanning is based on the person’s perception of his strength, comfort range,and the importance of staying along a particular path. After a direction isdetermined, he tries to move in that direction for a short distance with jointrates that maintain the body’s discomfort level below a particular thresh-old. Once the position is reached another direction is selected by balancingthe need to (cid:12)nish the task as directly as possible with restrictions derivedfrom the body’s limitations. Again, joint rates can be determined once a newdirection is established.The ob jective is to (cid:12)nd the tra jectories, both joint and end-e(cid:11)ector, thata human-like linkage would traverse to complete a lifting task The task canbe speci(cid:12)ed as a force that has to be overcome or imparted to reach a goalposition over an entire path. The task speci(cid:12)cation can be generalized todescribe a complicated task by letting the force be a function of body position,hand position, time, or other factors. In general, task speci(cid:12)cation can berepresented by a. In addition to task speci(cid:12)cation by a forceforce trajectory164CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORStra jectory, we have seen other behaviors in which human motion is guided byconstraints limiting joint and end-e(cid:11)ector tra jectories. Constraints that guidethe end e(cid:11)ector motion are,, and.discomfort levelperceived exertionstrengthDiscomfort levelis de(cid:12)ned in a mechanical sense. It is found by calculating,over the entire body, the maximum torque ratio: current torque divided bythe maximum torque at each individual joint for the current joint position andvelocity. Theis just 1. In general, when humans(cid:0) discomf ortComfort levelmove they try to maintain their e(cid:11)ort below a particular discomfort level.Therefore, it is desirable to dictate a motion that minimizes the maximumtorque ratio of a body in order to maximize the comfort level.Perceived exertionis a variable used to indicate the expected level of dif-(cid:12)culty in completing a task. It depends on the perception of the amount ofstrength required (an implicit function of the force tra jectory) and the amountof strength available. If perceived exertion is low then the discomfort level isnot expected to be exceeded for the paths \likely" to be taken to satisfy atask, especially for a path that travels a straight line between the initial bodyposition and the goal. However, if the perceived exertion is high, then theend-e(cid:11)ector path needs to deviate from a straight path in order to abide bythe comfort constraint. Perceived exertion is represented by a cone which isde(cid:12)ned by the maximum deviation angle of a path from its current position.Strength { the maximum achievable joint torque { also dictates end e(cid:11)ec-tor motion and path. For testing the strength-guided motion behavior anysuitable strength formulation would su(cid:14)ce; we used empirical data collectedby Abhilash Pandya of NASA Johnson Space Center [PMA91]. There are+two strength curves to represent the two muscle group strengths ((cid:13)exor andextensor) at each DOF.5.3.3 Motion ControlThe motion controller consists of three components (Figure 5.14):1. Condition Monitor which monitors the state of a body and suggestsmotion strategies.e(cid:11)ector will move.2. Path Planning Scheme (PPS) which plans the direction that an end-3. Rate Control Process (RCP) which determines the joint rates for motion.Thereports on the current state of a body: current posi-condition monitortion, maximum strength for a current position, current joint torques, etc. Itthen suggests motion strategies to thewhich determinespath planning schemean end-e(cid:11)ector’s direction of travel. The amount of motion of the end-e(cid:11)ectorin the suggested direction of travel can be arbitrarily set. The rate of travel,constrained by torque, for a path interval can then be computed by theratecontrol process. After the joint rates are resolved and new joint positions arefound, these procedures are repeated until the entire joint path is mapped5.3. STRENGTH GUIDED MOTION165force trajectory, comfort, 
perceived exertion, joint chainReach
Goal ?Behavioral  selectorstrengthexternal
dynamicsPath  Planning  Schemestrategy 1strategy 2strategy  n....Joint
ControlJoint
ControlJoint
ControlFigure 5.14: Strength Guided Motion Architecture.out in a manner that satis(cid:12)es the speci(cid:12)ed task. This system architectureis an iterative process which allows changes to the parameters at any timethrough other external processes. Possible situations to alter any of the pa-rameters are dropping or changing the mass of a load, redirecting the goal, orencountering an obstacle. This is di(cid:11)erent from the global nature of optimalcontrol-based algorithms. We handle similar global considerations through anexternal process [LWZB90].Condition MonitorThe condition monitor gathers information about the current state of a body,assembles the information, and suggests a motion strategy for the next pro-cedure to process. The motion strategies are a function of the constraintparameters: comfort, perceived exertion, and strength. Each motion strat-egy, based on the constraints, concentrates on a separate fundamental aspectof motion. The strategies can be divided into those that represent indirectjoint control and those that represent direct joint control. For indirect jointcontrol strategies, the end-e(cid:11)ector’s need to reach a goal is more importantthan joint considerations; and for direct joint control, joint considerations aremore important than reaching a goal. We can also interpret the strategies as166CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSparticular optimization problems. The condition monitor is the highest levelof the three procedures in predicting a path.Path Planning SchemeThe path planning scheme, guided by the condition monitor, determines thedirection to move. In general, the output of any system is bounded by its head-room: the available range of a variable within a constraint. In the case whenthere is much strength in a system (a situation where indirect joint controlapplies) the headroom can be used to suggest incremental joint displacements,d(cid:18). A larger headroom allows a larger displacement. The mapping betweenthe cartesian displacement and the joint displacement isdxJd(cid:18)=(517):whereis the 3 (cid:2)Jacobian matrix andis the number of joint displace-Jnnments. If the headroom for each joint is represented by a weighting vectorwproportional to, thend(cid:18)d^x=Jw(518):whereis a normalized direction of reach. The directionis then com-d^xd^xpared against a cone representing the feasible directions of travel derived fromperceived exertion. Ifis within the cone then the direction of motion shouldd^xbe, otherwise the direction can bepro jected onto the cone.d^xd^xWhen the system is relatively weak, the suggested direction of motionmust not violate the strength constraints. The decision process should shiftimportance from one strategy where the desirability to reach a goal is a ma jorcomponent of determining a suggested motion to an alternative strategy ofavoiding positions where the joints are greatly strained. This leads to schemeswhere direct joint control is imperative to avoid positions where joints arestrained [Lee92].Rate Control ProcessThe rate control process, the most basic of the three procedures, resolves thespeed with which a body moves along a prescribed end-e(cid:11)ector path. Thisrequires the use of dynamics, especially when the motion is fast. However,the incorporation of dynamics is di(cid:14)cult. When torques are speci(cid:12)ed to drivea motion (direct dynamics), control is a problem; when the driving forces arederived by kinematic speci(cid:12)cation (inverse dynamics), the forces are useful foronly a short time interval and they may violate the body’s torque capacity;and (cid:12)nally, when the forces optimize a particular function between two sets ofpositional constraints (boundary value problem), the method presumes thatthe optimization criteria is valid for the body’s entire range of motion.Dynamics equations can be interpreted as constraint equations solving forjoint tra jectories if they satisfy the conditions imposed by speci(cid:12)c end-e(cid:11)ectorpath and torque limits. The dynamics equations can be reformulated so that5.3. STRENGTH GUIDED MOTION167they provide a mapping between an end-e(cid:11)ector path and a binding torqueconstraint. A binding torque constraint is the maximum torque allowed todrive a body with maximum end-e(cid:11)ector acceleration without the end-e(cid:11)ectordeviating from the prescribed path. A greater torque would cause excessiveinertial force and therefore, undesirable path deviation. From the derivationof the reformulated dynamics equations originally derived to solve for pathcompletion in minimum time [Bob88], joint tra jectories can be found fromthe acceleration of an end-e(cid:11)ector.In addition to (cid:12)nding the tra jectories,the reformulated dynamic equations implicitly determine the force functions(joint torques) to guide an end-e(cid:11)ector along a speci(cid:12)ed path.Torque limits are established by the current discomfort constraint. Thediscomfort level variabledetermines the torque limit at each joint by adclsimple relation:dcl=(519):(cid:28)c;i(cid:28)(cid:18)()max;iwhereis the torque load for a particular joint. The value()isi(cid:28)(cid:18)max;i(cid:28)c;ithe maximum torque for the joint’s current position obtained by querying thestrength curves. When the value ofbecomes greater than one, there is nodclmore strength to accomplish a task and therefore the attempt to complete atask should cease. The discomfort level can be adjusted to achieve a desiredmotion. It in(cid:13)uences both the rate of task completion and the direction oftravel.5.3.4 Motion StrategiesThis is a catalogue of human motion strategies that are evaluated in thecondition monitor and are executed in the path planner. The strategies aregiven in the order of increasing discomfort.Available TorqueWhen a person moves, the tendency is to move the stronger joint. This issimilar to the forces due to a spring or other types of potential forces. Astronger spring, based on the spring’s sti(cid:11)ness coe(cid:14)cient, would yield a largerdisplacement per unit time than a weaker spring. Similarly, for a human, theamount of displacement for a joint depends not only on the strength that ajoint is capable of, but mainly on the amount of strength that is currentlyavailable. The amount of strength available, which is based on the di(cid:11)erencebetween the current required torque to support a particular position and thee(cid:11)ective maximum strength (the maximum strength factored by comfort), iscalled torque availability. If torque availability is low, motion should not beencouraged. Conversely, if the torque availability is high, the joint should domore of the work. Torque availability is the driving factor for a joint to moveand to thereby redistribute the joint torques so that the comfort level is moreuniform.168CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSReducing MomentAs a joint approaches its e(cid:11)ective maximum strength, the joint should move ina manner that avoids further stress (discomfort) while still trying to reach thegoal. A path towards the goal is still possible as long as the maximum strengthis not surpassed for any of the joints. As the body gets more stressed it shouldattempt to reduce the moment caused by a force tra jectory by reducing thedistance normal to the force tra jectory’s point of application. In addition, areduction in moment increases the torque availability of (at least) the jointthat is rapidly approaching its maximum strength. The reduction in thetotal moment involves examining the moments on a joint by joint basis. Ateach joint a virtual displacement is given to determine if that displacementprovides su(cid:14)cient moment reduction to continue moving in that direction.This strategy assumes that the body has enough e(cid:11)ective strength to allowits joints to move to positions where the overall stress level of the body issmaller than if the joints were only guided by kinematic demands.Pull BackThe two previous strategies depend on the current torque being less than themaximum strength. In these cases, maneuverability in torque space is highand therefore an end-e(cid:11)ector can still consider moving toward a goal withoutexceeding any joint’s maximum strength.When a particular joint reaches its maximum strength, however, then thatjoint can no longer advance toward a goal from the current con(cid:12)guration.The Pull Back strategy proposes that the end-e(cid:11)ector approach the goal fromanother con(cid:12)guration. In an e(cid:11)ort to determine another approach to the goal,the constraint of moving toward a goal within a restricted path deviation canbe relaxed. The emphasis of the strategy is one where the joints dictate animproved path in terms of torques. This can be accomplished by increasingthe ultimate available torque { the di(cid:11)erence of maximum strength to currenttorque { for a set of weak joints { joints that are between the joint which hasno ultimate available strength and an end-e(cid:11)ector.In general, the joint with the least amount of ultimate available torque willreverse direction and cause the end-e(cid:11)ector to pull back (move away from itsgoal). The idea is to increase the overall comfort level. When the joints forma con(cid:12)guration that has a greater level of comfort, there might be enoughstrength to complete the task. Then the governing strategy could return toReducing Moment, which allows the end-e(cid:11)ector to proceed toward the goal.The Pull Back strategy leads to a stable con(cid:12)guration. This is a posturethat a set of joints should form so that it can withstand large forces, suchas those caused when changing from a near-static situation to one that isdynamic.5.3. STRENGTH GUIDED MOTION169Added Joint, Recoil, and JerkWhen the three strategies, Available Torque, Reducing Moment, and PullBack have been exhausted and an agent still cannot complete a task, it isobvious that the active joints { the joints that were initially assigned to thetask { cannot supply su(cid:14)cient strength. When this occurs it should be deter-mined if the task should be aborted or if there are other means of acquiringadditional strength. One mode of acquiring more strength is to add a joint tothe chain of active joints. This assumes that the added joint is much strongerthan any of the active joints.Another mode to consider is to use the added joint to jerk { apply withmaximum force { the set of active joints. Jerk reduces the forces necessaryto complete a task for the set of active joints. Before jerking is initiated,a stable con(cid:12)guration should be formed by the active joints. After a stablecon(cid:12)guration has been formed and the added joint has jerked, the activejoints can then proceed to reach their goal since the required torques havedecreased. A third possibility is to recoil another set of joints and then jerkwith the recoiled set of joints in order to reduce the forces needed by the setof active joints to complete a task. For example, a weight lifter sometimesrecoils his legs and then pushs o(cid:11) to reduce the force required in his arms.5.3.5 Selecting the Active ConstraintsThe path determination process has been uncoupled into two active con-straints: comfort and perceived exertion. In the rate control process involvingdynamics, the two constraining parameters must be active to determine thejoint rates.comfort levels.At higher levels of control (such as in the path planner), both need notbe active simultaneously. In fact, as the torque levels change the applicabilityof a particular constraint to predict motion also changes. We use a modelthat relates the comfort level to the constraints. The strategies bound variousHigh Comfort.The perceived exertion constraint is not active but the com-fort constraint is, because any changes in acceleration (not necessarilylarge) may cause a joint to exceed the allowable discomfort level. Ingeneral, the force tra jectory associated with a motion of high comfortis negligible, but dynamics is important because of the relatively largeinertial e(cid:11)ects of the body. This group is bounded by motions that arecategorized by zero jerk condition [Gir91] and Available Torque.Regular Comfort.The end-e(cid:11)ector can advance toward the goal. Perceivedexertion and comfort are loosely constraining and dynamics should beevaluated. Available Torque and Reducing Moment bounds this comfortlevel.Discomfort.At this point the discomfort level for one or more joints aresurpassed. The perceived exertion constraint needs to be changed so170CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSthat a larger path deviation is allowed. Motion should have slowed downconsiderably, therefore dynamics is not important and, most likely, is notmeaningful. This group is formed by Reducing Moment and Pull Back.Intolerable DiscomfortMany of the joints’ comfort levels have been ex-ceeded and there may be other joints which could be approaching theirmaximum available torque. In such a situation, strategies can be com-bined. The pool of strategies are Added Joint, Pull Back, Recoil, andJerk. Perceived exertion is relaxed and depending on the combinationof the strategies, dynamics might be important.5.3.6 Strength Guided Motion ExamplesThe strategies for Available Torque, Reducing Moment, Pull Back, and AddedJoint are implemented in Jack. Figures 5.15, 5.16, 5.17, and 5.18 show thepaths that were produced from these conditions. The task is to place anincreasingly heavy load from various initial positions at a goal which is locatedabove the body’s head.In Figure 5.15, there are two curves which outline the path of the hand.The right curve is for a task that involves lifting a 10 pound ob ject; theleft curve is for a 20 pound ob ject. For the right curve, because the ob jectis relatively light, a fast motion is predicted and the solution resembles aminimum time path. For the left curve, the heavier weight draws the handcloser to the body. This path is rough because it is at the boundary of asolution determined by Available Torque and Reducing Moment. In Figure5.16, the right curve is for the 20 pound lift, and the left curve is for a liftof 30 pounds. Once again the algorithm predicts that a heavier ob ject wouldbring the hand closer to the body.Figure 5.17 shows the body with a heavier load (35 pounds). The bodyimmediately executes Pull Back. In this case, the body pulls back to a regionof high comfort and therefore the approach to the goal is smooth, without therough path evident in the previous (cid:12)gures. In Figure 5.18, the joint chain,initially composed of the joints between the hand and the shoulder, is allowedto extend to the waist. The algorithm decides that it is better to distributethe weight with more joints. Figure 5.18 shows the advantage of including thewaist in the set of active joints.These algorithms can be applied to any type of task, as long as it is force-based: even rising from a chair. A force tra jectory is used to represent thebody weight and the shoulder’s normal position when standing is used as theforce goal. The body leans forward to balance its weight (a consequence ofReducing Moment) to reach the shoulder goal.The average time of a path generation is under 10 seconds. Since ourexamples mainly involved heavy loads, static torque computations were used.The internal joint chain positions are determined by Jack inverse kinematics.5.3. STRENGTH GUIDED MOTION171Figure 5.15: Lifting a 20 pound and 10 pound Ob ject.Figure 5.16: Lifting a 30 pound and 20 pound Ob ject.172CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.17: Pull Back.Figure 5.18: Added Joint.5.3. STRENGTH GUIDED MOTION1735.3.7 Evaluation of this ApproachWe have tried to generate a realistic motion path for a slow, weight-liftingtask represented by a time- and position-dependent force tra jectory. Themethod maps out an entire path automatically and incrementally for a forcetra jectory over a set of constraints based on comfort level, perceived exertion,and strength. Because the body state is constantly updated, these constraintscan also be a function of an external model, such as fatigue.Motion is generated by integrating a condition monitor which suggestsbasic motion strategies, a path planning scheme which locally plans the end-e(cid:11)ector’s path and a rate control process which controls the joint rates. Thecondition monitor o(cid:11)ers strategies to pursue by balancing the goal of the taskagainst the resources that are currently available. The path planning schemeproposes a direction of travel by executing the basic strategies. The elusiveforce function that previous investigators have sought can be found by chang-ing the role of the dynamic equations to a constraint equation which is es-tablished with a dynamics model. By selecting the most restrictive constraintfrom the constraint equations, the maximum joint rates can be computed.Altering the constraints used in this problem still gives a goal-directedmotion that conforms to physical laws. We see this capability as an inherentlymodel-driven, (cid:13)exible, and crucial component for generating more naturalhuman behaviors. We will see shortly in Section 5.4 how this approach canbe extended to include collision avoidance during more complex actions.5.3.8 Performance GraphsThree types of graphs are developed for the user to analyze some pertinent me-chanical relations of the agent’s movements. The properties that are trackedand graphed are comfort levels, work, and energy (Figure 5.19, 5.20, and5.21).The histogram of comfort shows the current discomfort value as well asthe user-speci(cid:12)ed desired maximum. The e(cid:11)ective comfort level is the actualcomfort level that controls the movements. It includes factors that may a(cid:11)ectthe desired comfort level. Currently, an exponential function representingfatigue is one of the factors that can alter the e(cid:11)ective comfort level.The graph of work the agent performs at each iteration is computed as(cid:1)=jii(cid:1)W(cid:28)(cid:18)(520):ni=nXwhereis the number of DOFs,is the current torque of theth joint, andni(cid:28)i(cid:1)is the change in joint position ((cid:0)).(cid:18)i(cid:18)(cid:18)jj1(cid:0)The energy graph represents the sum of all the work performed by thejoints since the beginning of the task. Because the amount of energy that isexpended grows very fast, the curve that represents energy is automaticallyrescaled to (cid:12)t within the designated plot area when necessary.174CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigures 5.22, 5.23, and 5.24 show the comparative tra jectories of the handas load, comfort level, and perceived exertion are varied.5.3.9 Coordinated MotionCoordinated motion is when the execution of a task requires more than oneset of linkages either from the same agent or from two separate agents. Ourimplementation of coordinated motion follows the same parameter controlphilosophy. As in the task speci(cid:12)cation of a single linkage, a coordinated taskis made a function of comfort and path deviation.The coordination between the two joint chains is based on a master andslave relation: the weaker chain is the master and the stronger chain is theslave. The algorithm (cid:12)rst determines which is the weaker linkage (master),then its path is generated according to the strategy that is appropriate for itscomfort level. The other joint chain (slave) must comply with the master’snew position. Simple geometry from the master’s new position, the dimensionof the ob ject, and the magnitude of the slave’s incremental displacement isused to determine the slave’s new position (see Figure 5.25). The magnitudeof the master and slave’s incremental path is a nominal distance factoredby their respective comfort level; the master will travel less than the slave.The master-slave assignment switches whenever the relative comfort { thedi(cid:11)erence between the two linkages’ comfort level { reverses.The speci(cid:12)cation of a coordinated task begins by instantiating each ofthe joint chains as a separate task. Then the goal and the weight of theob ject to be moved is stated. Next, the kinematic relation between the twolinkages is speci(cid:12)ed; this involves identifying the attachment points on theob ject (handle) with the corresponding end-e(cid:11)ector. This is done for each ofthe chains that participates in the coordinated task. The end-e(cid:11)ectors areautomatically attached to the handle.Finally, two control parameters which control the characteristics of thepath need to be speci(cid:12)ed. The (cid:12)rst control parameter, al lowable comfortdi(cid:11)erence, speci(cid:12)es a tolerance in the relative comfort level before a master-slave switch is made. A graph can be called to display the di(cid:11)erence in comfortbetween the two joint chains. The other parameter controls the amount ofal lowable path deviation. The user can control the amount of path deviationby entering a ratio that determines the relative propensity between the pathof moving to the goal directly and the path that is sensitive to the agent’sstress level. Unlike a task involving a single chain, the motion of each chaincannot deviate or switch strategy without considerating the other chain. Thismeans that the strategies of the two linkages should coincide; both linkagesperforming Pull-Back would not be permitted. Because of this limitation, thestrategies available in this current implementation are only Available-Torqueand Reduce-Moment.5.3. STRENGTH GUIDED MOTION175Figure 5.19: Plots of Comfort Level, Work, and Energy (10 lb., pe = 0.30, cl= 0.50)176CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.20: Plots of Comfort Level, Work, and Energy (20 lb., pe = 0.30, cl= 0.50)5.3. STRENGTH GUIDED MOTION177Figure 5.21: Plots of Comfort Level, Work, and Energy (30 lb., pe = 0.30, cl= 0.50)178CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.22: 30 lb., pe = 0.30, cl = (0.40, 0.50, 0.85)Figure 5.23: 30 lb., pe = 0.30, cl = (0.40, 0.50, 0.75, 0.85)5.3. STRENGTH GUIDED MOTION179Figure 5.24: 5 lb., pe = (0.30, 0.20, 0.10, 0.05), cl = 0.50object's sphere
of possible virtual
positionsintersection
of both spheresslave's sphere
of possible virtual
displacementsvirtual
displacementobject's
virtual positionvirtual
displacementslavemasterFigure 5.25: Construction of the Path for Coordinated Motion.180CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORS5.4 Collision-Free Path and Motion Planning4Collision-free path planning has applications in a variety of (cid:12)elds such asrobotics task planning, computer aided manufacturing, human (cid:12)gure motionstudies and computer graphics simulations. A collision-free path for an ar-ticulated (cid:12)gure is the path along which the articulated (cid:12)gure moves froman initial con(cid:12)guration to a (cid:12)nal con(cid:12)guration without hitting any obstaclesresiding in the same environment as the articulated (cid:12)gure.A great deal of research has been devoted to the motion planning problemin the area of robotics within the last 10 years, e.g.[LPW79, LP87, LP83,Bro83b, BLP83, Bro83a, Don84, DX89, KZ86, Gou84]. However, despite theapplicability of motion planning techniques to computer graphics simulations,the problem has not been addressed much in the computer graphics commu-nity [Bre89, DLRG91].Articulated human (cid:12)gures are characterized by a branching tree structurewith many DOFs. Existing algorithms in robotics fall short in handling someof the issues encountered when dealing with these types of (cid:12)gures. We presentnovel algorithms that can address all these issues. The basic idea is thatinstead of treating all the DOFs in the (cid:12)gure together, we divide them upinto groups and treat these groups one by one and playback the path in acoordinated manner when all the groups are considered. Our motion planningsystem can also take into consideration the strength data of human (cid:12)gures sothat the planned motion will obey strength availability criteria (Section 5.3).5.4.1 Robotics BackgroundThe ma jor challenge of our problem is to handle a redundant branchingarticulated (cid:12)gure with many DOFs. Many of the robotics algorithms dealwith manipulators with relatively few DOFs, e.g. mobile robots which typ-ically have 3 DOFs or arm-like robots which have 6. Many of these algo-rithms are based on the use of the con(cid:12)guration space (C space) which isthe space encompassing the DOFs of the robot [LPW79, LP83]. The inher-ent di(cid:14)culty with this approach is due to the high dimensionality of the Cspace.It is well known that the worst case time bound for motion plan-ning for a robot arm is exponential in the dimensionality of its C space[SS83a, SS83b].It is only during the last few years that motion planningalgorithms that can handle manipulators with many DOFs have been pre-sented [BLL89b, BLL89a, BL89, Gup90, Fav84, Bre89].Very few studies consider articulated (cid:12)gures with branches. Barraquandet al gave an example involving a manipulator with 2 branches [BLL89b,BLL89a, BL89]. In their work, they create an arti(cid:12)cial potential (cid:12)eld in the3D workspace and the free path is found by tracking the valleys. A gain ine(cid:14)ciency is obtained as a result of the clever selection of potential functionsand heuristics. However, it is not clear how these can be selected in general.4Wallace Ching.5.4. COLLISION-FREE PATH AND MOTION PLANNING181Control  AlgorithmSequential
AlgorithmBasicAlgorithmPlayback
AlgorithmPlanar
AlgorithmFigure 5.26: Di(cid:11)erent Modules and their Associated Algorithms in the PathPlanning System.Faverjon et al [Fav84] partition the free space into oct-trees and uses someprobability measures to cut down the search tree during the A* search. Gupta[Gup90] handles sequential linkages with many DOFs using a sequential searchtechnique which basically treats the individual DOFs one by one instead ofconsidering all of them together. The initial stage of our path planner is basedon his work.5.4.2 Using Cspace GroupsThe main idea of our path planner is to handle the DOFs of the articulated(cid:12)gure not all at once but a certain number at a time. The general principle ofour path planner is (cid:12)rst to divide the DOFs into a number of groups which wecall Cspace groups . We then compute the collision-free motion for the DOFsin each group successively, starting from the (cid:12)rst group. After the motion ofthe DOFs in grouphas been planned, we parameterize the resulting motioniwith a parameter. The motion for the DOFs in group+ 1 will then betiplanned along this path by controlling the DOFs associated with it. Theproblem is then posed in a(cid:2)space if there areDOFs in this group.t(cid:18)kkWe proceed in this manner along the whole (cid:12)gure structure and solve for themotion for all the groups. Finally a playback routine is invoked to playbackthe (cid:12)nal collision-free path for the (cid:12)gure.Our system adopts a modular design in that it is made up of a number ofmodules each of which is based on an algorithm (Figure 5.26). Each modulecarries out a particular function and contributes to the whole path (cid:12)ndingprocess.On a more global perspective, the path (cid:12)nding procedure can be viewedas consisting of two phases: the computation phase and the playback phase.All of the steps involved in these phases are performed by the algorithmsdescribed in later sections.182CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSThe overall path planning procedure is outlined as follows:(cid:15)Phase:Computation1. Partition the articulated (cid:12)gure’s DOFs into Cspace groups accord-ing to a grouping scheme.2. Impose an order of traversal among the Cgroups. For a human(cid:12)gure, we use a depth-(cid:12)rst traversal. This means we plan themotion for one arm and then another.3. Invoke the control algorithm that handles traversal of the tree and(cid:12)nds the (cid:12)nal collision-free path. This algorithm will actually callupon a subsidiary algorithm, sequential algorithm , to compute thefree path along a branch of the tree structure. The sequentialalgorithm will in turn call another subsidiary algorithm, the basicalgorithm , to compute the path for the DOFs within each Cgroup.(cid:15)Phase:PlaybackAfter all the Cspace groups have been considered, a special playbackalgorithm will be called upon to traverse the tree structure in a reverseorder, collect and coordinate all the computed information and (cid:12)nallyplayback the overall collision-free path in discrete time frames. Thesetime frames can be further interpolated to produce smooth motion. Ide-ally the behavioral simulation loop controls the frame timing and hencethe production of the output postures.The translational movement of the articulated (cid:12)gure as a whole on a planecan also be generated with this planner. In this case, the (cid:12)gure resembles amobile robot with two degrees of translational freedom and one degree ofrotational freedom. The module that handles this case is named the PlanarAlgorithm .Figure 5.27 shows the general redundant branching articulated structureand symbols that we will use for reference. We will mainly focus on the upperbody of the human (cid:12)gure. The system can be easily applied to the legs toprovide stepping or foothold movements as we will show later.5.4.3 The Basic AlgorithmThe particular algorithm we have chosen is the one presented by Lozano-P(cid:19)erez in [LP87] due to its simplicity and intuitiveness. It (cid:12)rst constructs theC space for the articulated (cid:12)gure. For the sake of completeness, the processIf the manipulator haslinks, its con(cid:12)guration space can be constructednis described below.as follows:1.= 1.i2. While () doi < n5.4. COLLISION-FREE PATH AND MOTION PLANNING183b2bnnmlnb
1g1p(t12)
g12ln2ln1linksparameters(t)
11g11groupsl0m0l0302ll01b0branchesFigure 5.27: The Redundant Branching Articulated Figure.(a) Ignore links beyond link, and (cid:12)nd the ranges of legal values ofiqiby rotating linkaround the position of jointdetermined by theiicurrent value ranges ofand check for collision with theq; :::; q1i1(cid:0)surrounding obstacles. Self collision can be avoided by checkingcollision with linkages from the same (cid:12)gure as well. Mark thosejoint values at which linkwill have a collision as forbidden.i(b) Sample the legal range ofat the speci(cid:12)ed resolution.qi(c) Increment i and repeat step (a) for each of these value ranges.The free space is then represented by a regions data structure to explorethe connectivity between the cells. A graph is then built on these region nodesand an A* search is conducted to search for a free path from the start nodeto the goal node.5.4.4 The Sequential AlgorithmThe Sequential Algorithm handles the motion planning problem for the Cspacegroups along a sequential branch. This algorithm is based on but di(cid:11)ers fromGupta’s work.184CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSReferring to Figure 5.28, letbe the total number of Cspace groups onnthis branch. Let the joint DOFs associated with the groups be representedbywhereis the group number andis from 1 towhereis theqijijmimimaximum number of DOFs grouphas. Letbe the reference vertex foririgroup. It is basically the distal vertex of the link associated with the DOFsiin the group. Let() denote the tra jectory of the reference vertex. Theirtiriinitial and goal con(cid:12)gurations of the arm are given asand,=1..;=qqinjsgijij1..i.mThe algorithm is as follows:1. Compute a collision-free tra jectory for the links associated with group1. The tra jectory of the reference vertex on its link will be().rt12.= 2.i3. While ()i < nics.(c) Increment.i(a) along(), discretize the path according to a pre-speci(cid:12)ed reso-rti1(cid:0)lution. Compute a collision-free tra jectory for the DOFs in thethigroup fromtofor= 1..using the basic algorithm.iqqjmsgijij(b) given()()(), compute() using forward kinemat-qt; qt; :::; qt1j2jijrtiThe parameter used in parameterizing the path already computed can beeither interpreted as temporal or non-temporal. For a temporal interpretationof the parameter, the path computed has to be monotonic with respect tothe parametersimply because we cannot travel backward in time. Hencetbacktracking within the Cspace group is not allowed and the chance of (cid:12)ndinga path is greatly restricted. In the example shown in Figure 5.29, we will notbe able to come up with a path without backtracking. We have adopted a non-temporal interpretation of the parameter in most cases as this will increasethe chance of (cid:12)nding a path.Each C group deals with one parameter and a certain number of DOFs.The number of DOFs can vary between C groups so as to (cid:12)t into the structureof the (cid:12)gure. For example, the shoulder joint can be handled by one C groupwith 3 DOFs.The number of DOFs handled at a time also a(cid:11)ects the degree of optimalityof the resulting path (with respect to some criteria). Theoretically, the optimalpath can only be obtained by searching through the-dimensional C spacenbuilt from considering allDOFs together. However, such an algorithm hasnbeen proven to be exponential in the dimensionality of its C space [SS83a].There is a customary trade o(cid:11) between speed and optimality.Our choice of using the region graph instead of the visibility graph allowsfor the path to be positioned farther away from the obstacles, hence leavingmore room for the next linkage (Figure 5.30).5.4. COLLISION-FREE PATH AND MOTION PLANNING185Group n1rGroup 2Group 1
= 21mq1211qFigure 5.28: A Sequential Linkage.qtFigure 5.29: An Example Showing the Case that a Path can only be Foundwith Backtracking which Means the Parameter Takes on a Non-TemporalInterpretation.5.4.5 The Control AlgorithmThe control algorithm performs the entire path planning process:1. Apply the Planar Algorithm to the whole (cid:12)gure to obtain the planarcollision-free translational movement of the (cid:12)gure taken as a whole.2. Parameterize the resulting motion.3. Repeat the following to every branch, starting from the (cid:12)rst.(a) Apply the Sequential algorithm to the branch using the last pa-rameter in the (cid:12)rst group.(b) Parameterize the resulting path computed for this branch accord-ing to some prespeci(cid:12)ed resolution.(c) Invoke the PlayBack Algorithm to the branch to obtain the se-quence of joint angle values of the branch when moving along thecomputed path.(d) Record this sequence of joint angles in the array.F REEAN GLES186CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSq 11q( a )tt( b )Figure 5.30: (a) A Path that is Too Close to an Obstacle. This Leaves LittleRoom for the Next Linkage to Maneuver. (b) A Better Path that is FartherAway from the Obstacle.4. Apply the PlayBack Algorithm to the whole (cid:12)gure, starting from thelast group of the very last branch.5. The angle values obtained can then written into frames for continuousplayback.5.4.6 The Planar AlgorithmThe articulated (cid:12)gure can translate and rotate on a plane, navigating aroundobstacles. The whole (cid:12)gure behaves just like a mobile robot. The path plan-ning algorithm in this case deals with a three dimensional (2 translational, 1rotational) Cspace. We can handle this case simply with our basic algorithmor other existing mobile robot path planning techniques.5.4.7 Resolving Con(cid:13)icts between Di(cid:11)erent BranchesAlthough the di(cid:11)erent branches are attached to the same rear link of branchb0, we do not use the same parameterthat parameterizes the motion oftbranchin all these branches. The reason is that the parametersare in-tijb0terpreted as non-temporal in general. Hence, backtracking within the Cspacegroup is allowed and the values ofalong the computed path can be non-tijmonotonic. If we use the same parameter in computing the motion for the(cid:12)rst groups in all other branches, some of the joint angle values cannot be ob-tained uniquely during the (cid:12)nal playback phase. This reasoning may becomeclear after looking at the playback algorithm.Our solution to this problem is to further parameterize the already param-eterized path of the previous branch and then use the new parameterizationvariable in the next branch.5.4. COLLISION-FREE PATH AND MOTION PLANNING187Branch n, Group j-1Branch n, Group jqn, j-1Nqn, j(g)n, j-1a(g)
n, jA10tn, j-1LinearMapping0.710Na10.71n, jtqn, j-1( a )qn, jBranch n( b )Figure 5.31: An Example Showing How the Final Joint Angle Values of theWhole Figure are Obtained from the Cspace Associated with the CspaceGroups.5.4.8 Playing Back the Free PathDuring the playback phase, we start from the last group of branchandbnthen traverse the branches in a backward manner along branch,bb(cid:0)(cid:0)n1n2and so on and (cid:12)nally to branch. For example, let Figure 5.31 (a) representb0the con(cid:12)guration space for the last group of the last branch, i.e. groupgn;pnof branch. We then discretize the free path according to a pre-speci(cid:12)edbnplayback resolution. The number of discretization intervals for this last groupwill be equal to the number of time frames for the (cid:12)nal simulation.At every discretized point, say A, there is a corresponding () pair: theq; tqvalue is what we should set the last joint DOF to, and the parameteris usedtto deduce the motion of the preceding group. We (cid:12)rst set the last DOF to theqtvalue. Then we use the parameterin the pair to trace back to the preceding(proximal) group. Note that within this preceding group, the parameteristmonotonic by de(cid:12)nition. Hence we can uniquely determine the corresponding() pair within this preceding group. By the same token, we can continueq; ttracing back to the groups further preceding this one (Figure 5.31 (a)). Wecarry on in this fashion recursively until we come to the (cid:12)rst group withinthis branch.Note that at this point, all joint DOFs along this branch will have beenset to their correct value for this simulation time frame. The sequence of jointvalues along the free path for all the other branches should have also been188CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSrecorded in the array. The parameter value left unused willF REEAN GLESithen be used as an index into the recorded joint angle array and to uniquelydetermine the set of angles corresponding to the movement of the precedingbranch. The rest of the branches are processed similarly.We will examine the playback algorithm by looking into its two componentsseparately: the Final Playback algorithm and the Single Branch Single FramePlayback algorithm.The Final Playback AlgorithmThe Final Playback algorithm is driven by the behavior simulation cycle andgenerates the intermediate body postures culminating in the (cid:12)nal goal achieve-ment.(cid:15) Discretize the path computed for the last group in the last branch intoNdiscrete points according to some pre-speci(cid:12)ed resolution. This numberalso determines the total number of key postures or time frames we willgenerate for the (cid:12)nal simulation.(cid:15) For each time frame, do the following steps to get back the computedangles.1. Apply theSingle Branch Single Frame Playback Algorithmto branch, the last branch in the (cid:12)gure.bn2. A parameter value will be obtained at the termination of this algo-rithm. Use this parameter as an array index intoF REEAN GLESfor the next branch. The joint angles recorded for the next branchwill be read from the array element pointed to by this parameter3. Set the joint angles in the next branch to the values read from the4. The last parameter value read from the array is used to indexinto thearray for the next branch in a similarF REEAN GLESvalue.array.manner.5. Repeat the same process for the rest of the branches.6. Now all the joint angles in the articulated (cid:12)gure have been set totheir appropriate values in this time frame. What is left is theposition of the whole (cid:12)gure. The last parameter value obtainedfrom the last step is used to index into the path computed fromthe Planar Algorithm. Then we set the whole (cid:12)gure location tothat indexed position.The Single Branch Single Frame Playback AlgorithmLet the branch index we are considering be. Here branchhas a total ofiipigroups. This playback algorithm is called only after the motion for the5.4. COLLISION-FREE PATH AND MOTION PLANNING189last group in the branch is computed. This algorithm only deals with onediscretized point, and hence only one time frame.We start from the last group in the branch and go down the branch bydoing the following on each group.1. From the discretized point on the computed path, read the values ofthes associated with this Cspace group from the axes of the Cspace.qi;jThis is illustrated in Figure 5.31 (a) with a 2D Cspace as an example.2. Set the joints in the articulated chain corresponding to thesevariablesqto the values just found.3. Then read the normalized parameter valuefrom theaxis.ti;jt4. Through a linear mapping, obtain the corresponding discretized pointon the path computed for the next group down the branch from thisparameter value.Note that after this algorithm terminates, all the joint angles on thisbranch will be set to the appropriate values for this simulation time step.5.4.9 Incorporating Strength Factors into the PlannedMotionIn (Section 5.3) we demonstrated that realistic animation of lifting motionscan be generated by considering the strength of the human (cid:12)gure. The basicpremise of the method is that a person tends to operate within a comfortregion which is de(cid:12)ned by the amount of available torque. The static torquevalues at all the joints required to sustain a load is a function of (cid:12)gure con(cid:12)g-uration. The path planner incrementally updates the next joint angle valuesaccording to the available torque at the current con(cid:12)guration based on a num-ber of motion strategies.The planning methodology we have described so far divides the DOFs intogroups and plans the motion for each group sequentially. Therefore, only afterthe control algorithm terminates do we have the complete path for each DOF.However, we need to make use of the strength information during the planningprocess. This requires values of all joint angles at a certain con(cid:12)guration. Thesolution is to use the value of the current parameter and pro ject it over to therest of the angles that have not yet been computed. The pro jection functionP ROJis arbitrary (since this is only an approximation of reality), so we usejust a simple linear interpolation:P ROJt(cid:18)t(cid:18)(cid:18)initial() =+((cid:0)) .initialf inalSince our searching process uses the A* algorithm, a heuristic function isevaluated at every step to (cid:12)nd the node that has the smallest value. So farwe have been using only a spatial distance measure in this heuristic function.The path found will be optimal up to the size of the regions used. Now,190CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORShowever, we have a means to compute the required torque value of a partic-ular con(cid:12)guration, so this heuristic function can include terms representingstrength information. The weights attached to these terms represent the rel-ative importance of the quantities they represent. Possible terms to includeare:(cid:15). The total work done or energy expended can be measuredWork Doneby the term() (cid:1). The integration is done over the path taken.~~T(cid:18)d(cid:18)R(cid:15). The comfort level of the resulting motion can be measured byComfortthe available torque which is the quantity obtained by subtracting therequired torque from the strength limit at that particular joint con-(cid:12)guration. We can sum up all the contributions along the path asAvailT orque(cid:18)d(cid:18)() (cid:1)where the available torque is de(cid:12)ned in terms~~of its elements:RAvailT orque(cid:18)()=i~S tr(cid:18)T(cid:18)S tr(cid:18)> T(cid:18)()(cid:0)()ifii()i()i~~~~(cid:26)0otherwiseThe subscriptstands for the-th element in the vector.is theiiS trstrength limit vector. This integral value will then represent the overallcomfort level.future actions.This term will properly be useful only in thefunction as it only a(cid:11)ectsg(cid:15). Humans are not like robots: our strength will decrease withFatiguetime as a result of fatigue. We may include a term such ask()k~T(cid:18)dtto avoid taking a path that has a high torque value maintained over aRprolonged period of time.The path found by the collision-free path planner is the best up to thesize of the regions (the basic entities). Paths within regions are chosen by thestrength guided motion heuristics. For example, in Figure 5.32 (a), the leftpath may be chosen by search to be better than the one on the right. This pathcan then be further re(cid:12)ned by examining local comfort levels and invoking oneof the motion heuristics such as Available Torque, Reducing Moment and Pul lBack .5.4.10 ExamplesWe have experimented with a variety of reaches involving shelves and aper-tures. Figure 5.33 shows a human (cid:12)gure reaching through two apertures withboth arms. The path computed is collision-free and involves more than 20DOFs. These and similar examples take about 12 to 20 minutes elapsed timeto compute, depending on the complexity of the environment.Simulating the wide range of human motions requires a number of di(cid:11)erentbehavioral skills such as walking, grasping, and lifting. The path planner5.4. COLLISION-FREE PATH AND MOTION PLANNING191R
1R
2R
GR
S( b )R
1R
2R
1R
2R
GR
S( a )R
GR
S( c )Figure 5.32: (a) Part of a Sample Cspace Showing Two Possible Paths Leadingfrom the Start to the Goal Node. (b) The Corresponding Regions Graph. (c)Path Re(cid:12)nement after Considering Comfortable Motion Heuristics.interfaces with other existing techniques in simulating more complex humanbehaviors. Figure 5.34 shows a human (cid:12)gure climbing up a rocky surface.The climbing movement of each limb and the torso translation are producedby the path planner. Each limb is considered in turn by the planner; the otherthree limbs are (cid:12)xed by a point (reach) constraint. The spatial locations foreach hand grasp and foothold must be pre-speci(cid:12)ed by the user. Even thoughour path planner cannot handle closed loop systems directly, such motionscan be simulated with the help of other behaviors.5.4.11 Completeness and ComplexityThis planner is an approximate algorithm when backtracking among groupsis not employed: it may fail to discover a path even though one exists. Whenbacktracking is employed between groups, the algorithm is turned into a com-plete algorithm. Alternatively, the DOFs can be re-grouped di(cid:11)erently if apath cannot be found for the previous grouping scheme.The basic algorithm within a Cspace group has complexity(())Ormn(cid:0)k12whereis the number of DOFs,is the discretization interval,is thekrmnumber of faces and edges for the (cid:12)gure andfor the environment [LP87].nSince the number of DOFs in a Cspace group is bounded, the run time forthe basic algorithm can be treated as a constant. Consequently the wholealgorithm runs in() time whereis the total number of groups in the treeOpp192CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSstructure without backtracking between groups. With backtracking, the worstcase run time for the algorithm is exponential with respect to the number ofDOFs. This is the same as a conventional exhaustive search algorithm. Webelieve that the average run time of the algorithm is fast enough for practicalthough not interactive use.5.5 Posture Planning5Motion generation algorithms for geometric (cid:12)gures typically start with ini-tial, (cid:12)nal, and sometimes intermediate con(cid:12)gurations or postures. From theseinput postures, \natural" motion sequences are generated which satisfy givenoptimization criteria, e.g., time or energy. The assumption that the (cid:12)nal jointangle con(cid:12)guration of a highly redundant articulated (cid:12)gure such as the hu-man body is known { in advance { is rather unrealistic. Typically, only thetask-level goals of the end e(cid:11)ectors are known. In order for an agent to movein accordance with such task-level goals, we need to provide the agent withan ability to plan both intermediate and (cid:12)nal postures. Posture planning usesexplicit reasoning along with numerical procedures to (cid:12)nd a relatively naturaland collision-free sequence of postures that terminates at the desired goal.The inputs to the posture planner are positional or orientational goals forend e(cid:11)ectors. The posture planner (cid:12)nds the appropriate movements of rele-vant body parts needed to achieve the goals. It discovers a (cid:12)nal global posturesatisfying the given goals by (cid:12)nding intermediate motions that avoid collision.Typical motions used in the plan include stepping towards the workspace,spreading the legs, bending the upper body at the waist while the whole bodyremains balanced, and moving the upper arm with respect to the shoulder.Only the geometric aspects of body motion are considered at this time. Theagent is assumed to be initially located in the vicinity of the target ob ject sothat an end e(cid:11)ector goal can be achieved by taking only one or two steps.It is assumed that a given goal is not changed during the posture planningprocess. Collision-avoidance will be alluded to as necessary but the detailsare not addressed here [Jun92].The fundamental problem in achieving postural goals is controlling themassively redundant skeletal DOFs. There are 88 DOFs in the Jack model(not counting (cid:12)ngers). Figure 5.35 shows a simpli(cid:12)ed tree containing only36 DOFs that are necessary for gross motion. This excludes the vertebraewhich account for most of the remaining freedom. To solve the redundancyproblem, we should reduce the relevant DOFs to a manageable set at anygiven moment during motion planning. We use constraints to supplement thetraditional methods of robot motion planning in joint space done solely usingnumeric computations [LPW79, LP81, LP83, BHJ83, LP87, Bro83b, Bro83a,+Don87, Kha86, MK85, BLL89a, Bre89, CB92]. We reduce the complexity ofthe numeric computations by applying cues from qualitative knowledge andreasoning.5Moon Jung.5.5. POSTURE PLANNING193Figure 5.33: A Human Figure Reaching Through Two Apertures with BothArms.194CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.34: A Human Figure Climbing up a Rocky Surface. This Animationis Created with a Hybrid of Simulation Techniques.5.5. POSTURE PLANNING19520,21,2214,15,1617,18,192428,29,3011,12,13238,9,105,6,725,26,2743(34,35,36)21(31,32,33)Figure 5.35: The Reduced Degrees of Freedom of the Human Body.Here are the basic steps needed to determine postures needed to achievea given end e(cid:11)ector goal.1. Postulate primitive motions for the end e(cid:11)ector goal considering motiondependencies and a minimum disturbance constraint. For example, amotion dependency causes the palm to move forward and downwardwhen the upper body is bent forward about the waist. While decidingon movement of body parts, the body is sub jected to a constraint whichrequires minimal movements. For example, the agent does not bend ifa relevant ob ject can be grasped just by extending the hand. Primitivemotions are de(cid:12)ned in terms of salient parameters de(cid:12)ned in the 3DCartesian task space. These parameters are called control points andcontrol vectors. They are points and vectors de(cid:12)ned on parts of thebody that allow the conversion from task-level goals into an intuitivesubset of body DOFs.2. Use mental simulation to bind parameters of primitive motions and todetect collisions. The body is sub ject to inherent constraints such as onethat requires that the body remain balanced when a task is performed.3. If the mental simulation indicates that a body part would collide withan obstacle, identify collision-avoiding spatial constraints or suggest in-termediate collision-avoiding goals by reasoning about the qualitativegeometric relations among the body part, the goal placement, and theobstacle.196CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSforeheadview-vectorpalm-up-vectorpalm-centershouldertorso-up-vectorpelvis-centerpelvis-forward-vectorfootbody-ground-siteFigure 5.36: Task-Level Control Parameters: Control Points and Vectors.4. If the postulated motions violate already achieved goals or collision-avoiding constraints, use the amount of violation to replan and compen-sate for the violation.5.5.1 Functionally Relevant High-level Control Parame-tersTo control body postures e(cid:11)ectively, we represent spatial con(cid:12)guations of thebody in terms of \lumped" control parameters: control points and controlvectors (Figure 5.36). Control points are points de(cid:12)ned on important bodyparts such as feet, pelvis, head, and hands) to control their positions. Controlvectors are vectors de(cid:12)ned to control orientations. The task-space controlpoints or vectors are considered to be the reduced set of DOFs of the bodywith respect to the task-space. The number of task-space DOFs that weconsider is 14.A control point or vector moves relative to a base site of the control pointor vector. For example, the palm-center may be moved with respect to a basesite at the shoulder, the pelvis-center, or the body-ground-site. Or it may bemoved as the whole body moves with respect to the base site outside of thebody, that is, the origin of the world coordinate frame. The pelvis-forward-vector controls the horizontal orientation of the body and so is con(cid:12)ned torotate along the horizontal circle centered at the pelvis-center. The torso-5.5. POSTURE PLANNING197up-vector controls the forward/backward orientation of the upper body. Notethat the rough con(cid:12)guration of the body is de(cid:12)ned by the four primary controlparameters: body-ground-site, pelvis-center, pelvis-forward-vector, and torso-up-vector.5.5.2 Motions and Primitive MotionsMotions are de(cid:12)ned by specifying control parameters and the directions andthe distances to move them. We use the three standard directions along whichto move or rotate: sagittal (forward/backward), vertical (upward/downward),and sideward (leftward/rightward). These relative directions are consideredunit vectors de(cid:12)ned with respect to the body-centered coordinate frame.When we say that a control point moves forward, it means that the movementof the control point has the forward component. It may have other motioncomponents. For example, we use the motion goal move-by(forehead, down-ward, Dist) to refer to a goal of moving the forehead downward by distanceDist and orient-by(torso-up-vector, forward, Angle) to refer to a goal of ro-tating the torso-up-vector forward by angle Angle. A motion along a givendirection is decomposed into the three components along the standard direc-tions. This is convenient for two reasons. First, the component motions canbe independently planned, although interference among them must be takeninto account. Second, it is easy to (cid:12)nd cooperative motions that contributeto a given standard motion component.Note that motion move-by(forehead, downward, Dist) does not specifythe base site relative to which the forehead is to move. The forehead maymove relative to the pelvis-center (by bending the torso-up-vector forward),or relative to the body-ground-site (by lowering the pelvis-center down), or doboth in sequence or parallel. A motion which speci(cid:12)es the base site of a controlpoint or vector is called a primitive motion. Examples of primitive motionsare given in Table 5.1. Primitive move-by(left.palm-center, downward, Dist,pelvis-center) means that the agent moves the left palm center downward bydistance Dist with respect to base site pelvis-center. When the body-ground-site is moved, its base site is the world origin world. For example, move-by(body-ground-site, forward, Dist, world) means that the body-ground-site ismoved forward by distance Dist with respect to the world origin.The consequences of a primitive motion are not speci(cid:12)ed in terms ofprecondition-action-e(cid:11)ect rules. Rather they are computed by incrementalmotion simulation by taking advantage of the underlying geometric proper-ties of the body and the body balance constraint. Enumerating e(cid:11)ects ofprimitive motions under di(cid:11)erent conditions is simply not feasible in the con-tinuous domain of motion planning with massively redundant bodies.5.5.3 Motion DependenciesA positional goal is translated into a conjunction of simultaneous goals. Forexample, given a positional goal positioned-at(right.palm.center, GoalPos),198CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSTable 5.1: Primitive Motions for Manipulating Control Parameters.move-by(left.palm-center, downward, Dist)move-by(left.palm-center, downward, Dist, shoulder)move-by(left.palm-center, downward, Dist, pelvis-center)move-by(body-ground-site, forward, Dist, world)orient-by(pelvis-forward-vector, leftward, Ang)orient-by(pelvis-forward-vector, leftward, Ang, pelvis-center)orient-by(pelvis-forward-vector, leftward, Ang, body-ground-site)the di(cid:11)erence vector from the current position of the palm center to the goalposition GoalPos is decomposed into the forward, downward, and rightwardcomponent vectors, with their distances being F, D, and R respectively . Thatis, positional goal positioned-at(right.palm-center, GoalPos) is translated tothe conjunction of the three component motion goals: move-by(right.palm-center, forward, F), move-by(right.palm-center, downward, D), and move-by(right.palm-center, rightward, R).There are several ways to achieve each component motion goal. They aredetermined based on the motion dependencies among di(cid:11)erent body parts.More precisely, they are relationships among control parameters described asfollows:1. move-by(right.palm-center, forward, Dist) has four contributors:(a) move-by(right.shoulder, right.palm-center, forward, D1),(b) rotate-by(pelvis-forward-vector, leftward, Ang2),(c) rotate-by(torso-up-vector, forward, Ang3, pelvis-center),(d) move-by(body-ground-site, forward, D4, world),such that Dist = D1 + D2(Ang2) + D3(Ang3) + D4, where D2(Ang2)is the distance that the right-palm-center travels due to the rotationof the pelvis-forward-vector leftward by angle Ang2 and D3(Ang3) isthe distance that the right-palm-center travels due to the rotation ofthe torso-up-vector forward by angle Ang3. In general, these distancesare dependent on the body context, the situation in which the body isplaced. Hence it would be extremely hard to compute them analyticallyin advance without incremental simulation.2. move-by(palm-center, downward, Dist) has two contributors:(a) move-by(palm-center, downward, D1, shoulder),(b) move-by(shoulder, downward, D2)such that Dist = D1 + D2.5.5. POSTURE PLANNING1993. move-by(shoulder, downward, Dist) has two contributors:(a) rotate-by(torso-up-vector, forward, Ang1, pelvis-center),(b) move-by(pelvis-center, downward, D2, body-ground-site)such that Dist = D1(Ang1) + D2, where D1(Ang1) is the distancethat the shoulder travels due to the rotation of the torso-up-vector withrespect to the pelvis-center by angle Ang1.4. move-by(palm-center, leftward, Dist) has four contributors:(a) move-by(right.shoulder, right.palm-center, leftward, D1),(b) rotate-by(pelvis-forward-vector, leftward, Ang2),(c) rotate-by(torso-up-vector, leftward, Ang3, pelvis-center),(d) move-by(body-ground-site, leftward, D4, world),such that Dist = D1 + D2(Ang2) + D3(Ang3) + D4, where D2 is thedistance that the right-palm-center travels due to the rotation of thepelvis-forward-vector leftward by angle Ang2 and D3 is the distancethat the right-palm-center travels due to the rotation of the torso-up-vector leftward by angle Ang3.How much to move or rotate the contributing control parameters to achievea given goal is not part of what we call motion dependencies. They aredetermined by mental simulation.5.5.4 The Control Structure of Posture PlanningThe control structure of posture planning (Figure 5.37) is described. Theplanner maintains the partial plan (the plan under construction that has un-ordered goals and motions) and the list of constraints on goals or motions.The partial plan is initialized to the input goals: [positioned-at(palm-center,GoalPos), aligned-with(palm-up-vector, GoalVector)], which are sub ject to theproximity constraints. Theis invoked to suggest a se-motion postulatorquence of primitive motions needed to achieve the current goal. The motionsare (cid:12)rst postulated without considering collisions, while respecting the previ-ously identi(cid:12)ed constraints. When the motions are postulated, themotionsimulatorselects motions to achieve the positional goal. To do so, the mo-tion dependencies among body parts are used. They permit many alternativeways of achieving a given goal. They are partly constrained by the minimumdisturbance constraint which prescribes that body parts farther away from theend e(cid:11)ector or the upper body are to move only when movements of bodyparts nearer the end e(cid:11)ector or the upper body are not su(cid:14)cient to achievea given postural goal. The minimum disturbance constraint is procedurallyincorporated into the motion postulator. To determine the values of distanceparameters, selected motions are simulated by incrementally changing the200CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORScollision-avoiding goal generatedresult of simulationstartProximity
-
Goal
GeneratorMotion
PostulatorMotion
Simulatorpostulated
motionscurrentmotionsend-effectorgoalcurrent
goals
/
constraintscollisionCollision
Avoiding
Goal
GeneratorSimulated
Geometric
WorldPARTIAL PLAN / CONSTRAINT LISTcollision-avoiding goals / constraints:  Data Flow:Control  FlowFigure 5.37: The Overall Control of Planning.control parameters until the goal values are reached. The incremental stepsize for each control parameter is assumed to be (cid:12)xed.When the motions are determined, theis again in-motion simulatorvoked to detect potential collisions of important body parts, hand(s), theupper body (including the head), and the lower body. Collision detection ofimportant body parts seems su(cid:14)cient for the purpose of (cid:12)nding a gross motionplan. If collisions are detected, themarks the faces of themotion simulatorobstacles with which body parts collided for the (cid:12)rst time and continues thesimulation, even with ob ject penetration, until the selected motions are fullyachieved. This reveals all potential body part collisions and provides morelookahead without signi(cid:12)cant extra cost. If collision has occurred during themotion simulation the planner calls thecollision-avoiding goal genera-torto generate intermediate goals. The achievement of the collision-avoidinggoals is then planned by themotion postulator.5.5.5 An Example of Posture PlanningThe process of posture planning is best described using an example. Con-sider a positional goal positioned-at(right.palm-center, GoalPos). Assume thatthe proximity constraints which the positional goal is sub ject to are already5.5. POSTURE PLANNING201achieved and are maintained. For simplicity of exposition, the end e(cid:11)ectorgoal is assumed to have only the forward and downward components, ignoringthe sideward component. Consider the situation in Figure 5.38.1. Translate the input goal positioned-at(right.palm-center, GoalPos) toits contributor motions: move-by(right.palm-center, forward, F) andmove-by(right.palm-center, downward, D), where the goal vector fromthe palm center to goal position GoalPos is decomposed to vectors Fand D.2. According to the minimum disturbance constraint, the shoulder is tried(cid:12)rst as the base joint of the palm-center motions. This will be done ifthe distance between the shoulder and the goal position is less than thearm length. Otherwise, the shoulder should be (cid:12)rst moved so that thedistance between the new shoulder position and the goal position willbe less than the arm length. A plausible goal position of the shouldercan be simply found as shown in Figure 5.39. To (cid:12)nd it, (cid:12)rst get thevector ArmLine between the shoulder and the goal position of the ende(cid:11)ector. Get the vector Arm whose length is equal to the arm lengthand which is parallel to vector ArmLine. Then the di(cid:11)erence vectorDi(cid:11) = ArmLine (cid:0) Arm becomes the motion vector of the base site, theshoulder. Suggest parallel primitive motions move-to(right.palm-center,forward, F0, right.shoulder) and move-to(right.palm-center, downward,D0, right.shoulder) and add them to the goal tree. Then, extract thedownward and forward motion components from the di(cid:11)erence vectorDi(cid:11). Let the downward component be DDist and the forward compo-nent FDist (Figure 5.39). Add two goals move-by(right.shoulder, down-ward, DDist) and move-by(right.shoulder, forward, FDist) to the goaltree. They are parallel goals to be achieved at the same time. They areadded to the goal tree to be achieved before the end e(cid:11)ector motionspreviously added.3. Get the top two goals from the goal tree. Nonlinear planning is used toplan for the goals, that is, each goal is planned for independently start-ing from the current situation. If the goals interfere, they are resolvedby (cid:12)nding and removing the causes of the con(cid:13)ict. According to the mo-tion dependencies, goal move-by(right.shoulder, forward, FDist) can beachieved by (i) orient-by( torso-up-vector, forward, Ang2, pelvis-center)and (ii) move-by(body-ground-site, forward, F1, world) where FDist =F1 + F2(Ang2). Goal move-by(right.shoulder, downward, DDist) isachieved by (a) orient-by(torso-up-vector, forward, Ang1, pelvis-center)and (b) move-by(pelvis-center, downward, D2, body-ground-site), whereDDist = D1(Ang1) + D2. Here D1(Ang1) is the movement of the shoul-der caused by bending the torso-up-vector by angle Ang1 and D2 is themovement of the shoulder caused by lowering the pelvis-center.4. Consider subplan (i) orient-by(torso-up-vector, forward, Ang1, pelvis-center) and (ii) move-to(body-ground-site, forward, FDist, world) for202CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSthe forward component goal of the shoulder. Based on the minimumdisturbance constraint, motion (i) alone is (cid:12)rst tried, because its basesite pelvis-center is more proximal to the control point, the shoulder,than is the base site (world) of motion (ii). Motion (i) orient-by(torso-up-vector, forward, Ang2, pelvis-center) helps achieve the (cid:12)rst goal aslong as the precondition holds that the shoulder is above the horizon-tal plane passing through the pelvis-center (Figure 5.40). Incrementallysimulate motion (i) to determine angle parameter Ang2 (Figure 5.39).This angle value is di(cid:14)cult to predict without incremental simulation,because this value is a(cid:11)ected by the body balance constraint. Simulatethe motion until the forward component goal of the shoulder is achievedor the precondition is violated. Because the precondition is violatedbefore the forward component goal is achieved, it is concluded that mo-tion (i) alone is not su(cid:14)cient. Motion (ii) is used to (cid:12)ll the remaininggap. The distance F1 that motion (ii) should achieve is easily predictedby simple arithmetic (Figure 5.41). The temporal relation between mo-tions (i) and (ii) is determined, because the motion whose base site isthe world origin world is to be achieved (cid:12)rst.Consider subplan (a) orient-by(torso-up-vector, forward, Ang1, pelvis-center) and (b) move-by(pelvis-center, downward, D2, body-ground-site),where DDist = D1(Ang1) + D2, for the downward component goal ofthe shoulder. According the minimum disturbance constraint, motion(a) alone is chosen (cid:12)rst, because its base site pelvis-center is more prox-imal to the hand than is the base site of motion (b), body-ground-site.This is compatible with the observation that lowering the pelvis-centerrequires bending the knees and is considered more di(cid:14)cult than bend-ing the torso-up-vector. (This may not be possible if bending the torsocauses collision. But in this example, collisions are not considered.)Incrementally simulate motion (b) to determine angle parameter Ang2.Simulate until the downward component goal of the shoulder is achieved,that is, DDist = D1(Ang1). Because motion (a) for the downward com-ponent is the same (bending the torso) as motion (ii) for the forwardcomponent, the simulation of motion (a) will be done by augmenting orsubtracting the simulation of motion (ii).5. The above reasoning suggests two motions move-by(body-ground-site,forward, F1, world) and orient-by(torso-up-vector, forward, Ang2, pelvis-center) for the forward component goal of the shoulder, and one motionorient-by(torso-up-vector, forward, Ang1, pelvis-center) for the down-ward component goal of the shoulder. If the two sets of motions do notinterfere with each other with respect to the achievement of the two par-allel goals, the motion postulation is successful. But this is not the case.When motion orient-by(torso-up-vector, forward, Ang1, pelvis-center) issimulated to achieve the downward motion component DDist, the shoul-der goes below the horizontal plane passing through the pelvis-center(Figure 5.42). It therefore also contributes negatively to the forward mo-5.5. POSTURE PLANNING203tion component FDist, partly violating the goal move-by(right.shoulder,forward, FDist). The(cid:12)rst attempts to resolve thismotion postulatorgoal interference by modifying distance parameters of motions in the cur-rent partial plan (goal tree). (If the rebinding of motion parameters fails,thewill try other combinations of contributing mo-motion postulatortions.) When themodi(cid:12)es the subplans, subplansmotion postulatorwith more contributing motions are tried earlier than are those withfewer motions. In particular, subplans with single motions cannot berebound without destroying the goals they achieved. Consider a subplanwith more than one contributing motion. If the distance of one motionis modi(cid:12)ed to avoid the goal interference, this also causes the violationof the goal of the subplan. But the distances of the other motions can bemodi(cid:12)ed to compensate for that violation. In the current example, themotion postulator keeps the subplan for the downward goal of the shoul-der, because it has only one motion orient-by(torso-up-vector, forward,Ang2, pelvis-center). This decision a(cid:11)ects motion orient-by(torso-up-vector, forward, Ang1, pelvis-center) in the subplan for the forward goalof the shoulder, with Ang1 := Ang2. (This decision in fact turns out tobe the same as the initial arbitrary decision used to simulate the twosubplans.) Simulate the two subplans thus obtained. Get the distanceof the negative contribution from the resulting simulated situation. Letthe distance be NegDist. The distance can be achieved by adding anadditional motion move-by(body-ground-site, forward, NegDist, world).This motion compensates for the distance DegDist without doing anyharm to the downward goal of the shoulder. But themotion postula-toris supposed to see (cid:12)rst that modifying the distance parameters ofthe current plans will work, before trying to add other motions. Thatis possible in this case by modifying motion move-by(body-ground-site,forward, FDist, world) to move-by(body-ground-site, forward, FDist +NegDist, world).6. After the shoulder goals are achieved, the current motions in the goaltree to be considered are the downward and forward motions of thepalm-center with respect to (the current position of ) the shoulder. Sim-ulate these motions. Both motion components are achieved with equalimportance given to each of them. They will be achieved if there areno hindrances (constraints or obstacles) at all, because the position ofthe shoulder has been obtained based on that assumption.If thereare hindrances, the base of the palm-center (the shoulder) needs to berelocated as the palm-center is moved to intermediate goals. The in-termediate goals are found to avoid the hindrances while achieving thegoal of the palm-center.204CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFigure 5.38: The Initial Situation for a Given Goal.FDistshoulderDDistnew
shoulderFigure 5.39: The Two Components of the Shoulder Di(cid:11)erence Vector.hand5.5. POSTURE PLANNING205The  horizontal  linepassing the pelvis centerFigure 5.40: As Long as the Shoulder is Above the Horizontal Line, Bendingthe Torso-Up-Vector Moves the Shoulder Forward.shoulderFDistAng2DDistF2F1( FDist = F1 + F2(Ang2) )Figure 5.41: Achieving the Forward Component: By Moving the Body-Ground-Site and Bending the Torso.206CHAPTER 5. SIMULATION WITH SOCIETIES OF BEHAVIORSFDistAng1BDistDDist(DDist = D1(Ang1) )BDist : Negative  Contribution
of  Bending  the TorsoFigure 5.42: Achieving the Downward Component: By Rotating the Torso-Up-Vector Forward, Which Violates the Forward Component by DistanceBDist.Chapter 6Task-Level Speci(cid:12)cationsSo far we have been talking about real-time interactive display and manip-ulation of human (cid:12)gures, with the goal of enabling human factors engineersto augment their analyses of designed environments by having human (cid:12)gurescarry out tasks intended for those environments. This chapter explores theuse of task-level speci(cid:12)cations as an alternative to direct manipulation forgenerating task simulations.By now, the reader should be convinced of the value of being able to sim-ulate, observe and evaluate agents carrying out tasks. The question is whatis added by being able to produce such simulations from high-level task spec-i(cid:12)cations. The answer is e(cid:14)cient use of the designer’s expertise and time. Adesigner views tasks primarily in terms of what needs to be accomplished, notin terms of moving ob jects or the agent’s articulators in ways that will eventu-ally produce an instance of that behavior { e.g., in terms of slowing down andmaking a left turn rather than in terms of attaching the agent’s right hand tothe clutch, moving the clutch forward, reattaching the agent’s right hand tothe steering wheel, then rotating the wheel to the left and then back an equaldistance to the right. As was the case in moving programming from machine-code to high-level programming languages, it can be more e(cid:14)cient to leaveit to some computer system to convert a designer’s high-level goal-orientedview of a task into the agent behavior needed to accomplish it. Moreover,if that same computer system is (cid:13)exible enough to produce agent behaviorthat is appropriate to the agent’s size and strength and to the particulars ofany given environment that the designer wants to test out, then the designeris freed from all other concerns than those of imagining and specifying theenvironments and agent characteristics that should be tested.This chapter then will describe a progression of recent collaborative ef-forts between the University of Pennsylvania’s Computer Graphics ResearchLab and the LINC Lab (Language, INformation and Computation) to movetowards true high-level task speci(cid:12)cations embodying the communicative rich-ness and e(cid:14)ciency of Natural Language instructions.The (cid:12)rst section will describe our earliest work on using simple Natural207208CHAPTER 6. TASK-LEVEL SPECIFICATIONSLanguage commands to drive task animation. This work also dealt with anaspect of performance simulation that is rarely communicated in task spec-i(cid:12)cations: how long an action will take an agent to perform. Section 6.2describes our next e(cid:11)ort at using Natural Language commands to drive taskanimation, focusing on how kinematic and spatial aspects of desired behaviorare conveyed in Natural Language commands.One of the consequences of these (cid:12)rst two studies is understanding thevalue of a strati(cid:12)ed approach to mapping from language to behavior: it is note(cid:14)cient for, say, the language understanding components to make decisionsthat commit the agent moving a hand or a knee in a particular way, unlessthose movements are stated explicitly (but rarely) in the text. Because ofthis recognized need for intermediate representations between Natural Lan-guage descriptions and animation directives, an experiment was performed,described in Section 6.3, in which an intermediate level compositional lan-guage was created for specifying task-actions and generating task-level ani-mated simulations from scripts written in this language. This demonstrationpaves the way for the ultimate connection between the behavioral simulationstructure of the preceding Chapter and the conceptual structures of this one.Each of these early e(cid:11)orts focused on individual commands in NaturalLanguage. Task speci(cid:12)cations, on the order of Natural Language instructions,go beyond individual commands in specifying what an agent should (andshouldn’t) do. Since our current work is aimed at generating task animationsfrom speci(cid:12)cations as rich as Natural Language instructions, we devote thediscussion in Section 6.4 to describing some features of instructions and whatan understanding system requires in order to derive from them what a personwould in understanding and acting in response to instructions.6.1 Performing Simple Commands1Our (cid:12)rst work on driving task animation through Natural Language com-mands was a prototype system developed by Je(cid:11)rey Esakov that exploredsimple relations between language and behavior [EBJ89]. In this case, the de-sired behaviors were simple arm reaches and head orientation (view) changeson the part of the animated (cid:12)gures. While seemingly very easy, these tasksalready demonstrate much of the essential complexity underlying language-based animation control.6.1.1 Task EnvironmentIn Esakov’s work, the tasks to be animated center around a control panel (i.e.a (cid:12)nite region of more or less rigidly (cid:12)xed manually-controllable ob jects) {here, the remote manipulator system control panel in the space shuttle withits variety of controls and indicators. Because Esakov was producing taskanimations for task performance analysis, he needed to allow performance1Je(cid:11)rey Esakov.6.1. PERFORMING SIMPLE COMMANDS209to depend upon the anthropometry of the agent executing the task. In theexperiments, all the controls were in fact reachable without torso motion bythe agents being animated: failure situations were not investigated and thefully articulated torso was not yet available. An animation of one of theexperiments can be found in [EB91].6.1.2 Linking Language and Motion GenerationThe primary focus of this work was to combine Natural Language task speci-(cid:12)cation and animation in an application-independent manner. This approachused the following Natural Language script:John is a 50 percentile male.Jane is a 50 percentile female.John, look at switch twf-1.John, turn twf-1 to state 4.Jane, look at twf-3.Jane, look at tglJ-1.Jane, turn tglJ-1 on.John, look at tglJ-2.Jane, look at twf-2.Jane, turn twf-2 to state 1.John, look at twf-2.John, look at Jane.Jane, look at John.(The abbreviations denote thumbwheels such asand toggle switches suchtwf-1as. Thumbwheels have states set by rotation; toggles typically havetglJ-1two states,or.)onoffThis type of script, containing simple independent commands, is com-mon to checklist procedures such as those done in airplanes or space shuttles[Cen81]. The verb \look at" requires a view change on the part of the (cid:12)gure,and the verb \turn" requires a simple reach. (Fine hand motions, such as(cid:12)nger and wrist rotations, were not animated as part of this work.) The twoprimary problems then are specifying reach and view goals, and connectingob ject references to their geometric instances.6.1.3 Specifying GoalsFor these reach tasks the goal is the 3D point which the (cid:12)ngertips of the handshould touch. A view goal is a point in space toward which one axis of anob ject must be aimed. Spatially, such goals are just Peabody sites and mustbe speci(cid:12)ed numerically with respect to a coordinate system. Within thenatural language environment, goals are not seen as coordinates, but ratheras the ob jects located there { for example,John, look at switch twF-1.Jane, turn switch tglJ-1 on.210CHAPTER 6. TASK-LEVEL SPECIFICATIONSBecause the exact locations of the switches is unimportant at the languagelevel, in creating an animation, the switch namemust be mapped totglJ-1the appropriate switch on the panel in the animation environment. The sameprocess must be followed for the target ob ject toward which an ob ject axismust be aligned in a view change. This problem reduces to one of ob jectreferencing.6.1.4 The Knowledge BaseIn general, all ob jects have names. Since the names in the task speci(cid:12)cationenvironment may be di(cid:11)erent from those in the animation environment, theremust be a mapping between the names. The knowledge base that Esakov usedcontained information about ob ject names and hierarchies, but not actualgeometry or location. He used a frame-like knowledge base called DC-RL tostore symbolic information [Ceb87]. For example, the DC-RL code for anisolated toggle switch,, follows:tglJ-1To reference this switch from within the animation environment, a map-ping (cid:12)le was generated at the same time the graphical ob ject was described.{ concept tglJ-1 from controlhaving ([role name with [value = "TOGGLE J-1"]][role locative with [value = panel1]][role type-of with [value = switch]][role sub-type with [value = tgl]][role direction with [value = (down up)]][role states with [value = (off on)]][role movement with [value =(discrete mm linear ((off on) 20 5))]][role current with [value = off]])}{ concept ctrlpanel from panelfighaving ([role twF-1 with[ value = ctrlpanel.panel.twf_1 ]][ value = ctrlpanel.panel.twf_2 ]][ value = ctrlpanel.panel.twf_3 ]][role twF-2 with[role twF-3 with[role tglJ-1 with[role tglJ-2 with[ value = ctrlpanel.panel.tglj_1 ]][ value = ctrlpanel.panel.tglj_2 ]])}6.1. PERFORMING SIMPLE COMMANDS211The names,,correspond to the names of switches intwF-1twF-2tglJ-1the existing knowledge base panel description called. These namespanelfigare mapped to the corresponding names in the animation environment (e.g.,ctrlpanel.panel.twf 1, etc.) and are guaranteed to match.6.1.5 The Geometric DatabaseThe underlying geometric database is just Peabody code. The salient tog-gle and thumbwheel locations were simply mapped to appropriate sites on ahost segment representing the control panel ob ject. The relevant part of thePeabody description of the panel (cid:12)gure is shown:figure ctrlpanel {segment panel {psurf = "panel.pss";site base->location =trans(0.00cm,0.00cm,0.00cm);site twf_1->location =trans(13.25cm,163.02cm,80.86cm);site twf_2->location =trans(64.78cm,115.87cm,95.00cm);site twf_3->location =trans(52.84cm,129.09cm,91.43cm);site tglj_1->location =trans(72.36cm,158.77cm,81.46cm);site tglj_2->location =trans(9.15cm,115.93cm,94.98cm);}}switches.This entire (cid:12)le is automatically generated by a modi(cid:12)ed paint program. Usingthe panel as a texture map, switch locations are interactively selected and thecorresponding texture map coordinates are computed as the site transforma-tion. The panel itself is rendered as a texture map over a simple polygonand the individual sites then refer to the appropriate visual features of the6.1.6 Creating an AnimationLinking the task level description to the animation requires linking both ob-ject references and actions. A table maps the names of ob jects from the taskdescription environment into the psurf geometry of the animation environ-ment.In this simple problem domain the language processor provides theother link by associating a single key pose with a single animation command.Each part of speech (cid:12)lls in slots in an animation command template. SimpleJack behaviors compute the (cid:12)nal posture required by each command whichare then strung together via simple joint angle interpolation.212CHAPTER 6. TASK-LEVEL SPECIFICATIONS6.1.7 Default Timing ConstructsEven though the basic key poses can be generated based upon a NaturalLanguage task description, creating the overall animation can still be di(cid:14)cult.We have already discussed posture planning and collision avoidance issues, butthere is yet another problem that bears comment. From the given commandinput, the timing of the key poses is either unknown, unspeci(cid:12)ed, or arbitrary.Action timings could be explicitly speci(cid:12)ed in the input, but (language-based) task descriptions do not normally indicate time. Alternatively, de(cid:12)ningthe time at which actions occur can be arbitrarily decided and iterated untila reasonable task animation can be produced. In fact, much animator e(cid:11)ortis normally required to temporally position key postures. There are, however,more computational ways of formulating a reasonable guess for possible taskduration.Several factors e(cid:11)ect task performance times, for example:level of ex-pertise, desire to perform the task, degree of fatigue (mental and physical),distance to be moved, and target size. Realistically speaking, all of these needto be considered in the model, yet some are di(cid:14)cult to quantify. Obviously,the farther the distance to be moved, the longer a task should take. Further-more, it is intuitively accepted that performing a task which requires precisionwork should take longer than one not involving precision work: for example,threading a needle versus putting papers on a desk.Fitts [Fit54] and Fitts and Peterson [FP64] investigated performance timewith respect to two of the above factors, distance to be moved and target size.It was found that amplitude (, distance to be moved) and target width ()AWare related to time in a simple equation:Movement Time =+logab(61):2AWwhereandare task-dependent constants. In this formulation, an index ofabmovement di(cid:14)culty is manipulated by the ratio of target width to amplitudeand is given by:ID = log(62):2AWThis index of di(cid:14)culty shows the speed and accuracy tradeo(cid:11) in movement.Sinceis constant for any particular task, to decrease the performance timeAthe only other variable in the equationmust be increased. That is, theWfaster a task is to be performed, the larger the target area and hence themovements are less accurate.This equation (known as Fitts’ Law) can be embedded in the animationsystem, since for any given reach task, bothandare known. The con-AWstantsandare linked to the other factors such training, desire, fatigue,aband body segments to be moved; they must be determined empirically. Forbutton tapping tasks, Fitts [FP64] determined the movement time () toM TbeM TIDmsec= 74(cid:0) 70(63):arm6.1. PERFORMING SIMPLE COMMANDS213Task Duration Times (msec)ActorActionID Fitts DurationScaled DurationJohnLook twf-12.96321963JohnTurn twf-15.473351004JohnLook tglJ-24.195661697JohnLook twf-24.015301590JohnLook Jane4.646551966JaneLook twf-34.28584876JaneLook tglJ-13.64456685JaneTurn tglJ-15.39329493JaneLook twf-24.16560840JaneTurn twf-24.99299449JaneLook John4.33594891Table 6.1: Task Durations Using Fitts’ Law.In determining this equation, it was necessary to (cid:12)lter out the extraneousfactors. This was done by having the sub jects press the button as quicklyas possible and allowing them to control the amount of time between trials.Jagacinski and Monk [JM85] performed a similar experiment to determine themovement time for the head and obtained the following equationMT= 199(cid:0) 268(6.4)IDmsechead0ID= log(6.5)02AWW(cid:0)0This equation is the result of equating the task to inserting a peg of diameterW0Winto a hole of diameter, and resulted in a better (cid:12)t of the data.For our purposes the above constants may not apply. Since it was ourdesire to have the man in our animation move sluggishly and the woman movequickly (but not too quickly), we scaled Equations 6.3 and 6.4 by di(cid:11)eringconstants.M TM T= 3 (cid:3)armman(arm)M TM T= 3 (cid:3)man(head)headM T:M T= 15 (cid:3)armwoman(arm)M T:M T= 15 (cid:3)woman(head)headThis width of the target,in equation 6.2 was chosen to be 1cm. ForW(cid:14)head movements, we chose=33after [JM85]. This results in the actionW:0durations shown in Table 6.1.Although Fitts’ Law has been found to be true for a variety of movementsincluding arm movements (= 5 (cid:0) 30cm), wrist movements (= 13cm)AA:214CHAPTER 6. TASK-LEVEL SPECIFICATIONS[Dru75, JM85, LCF76], and head movements (= 245 (cid:0) 750) [JM85] theA::(cid:14)application to 3D computer animation is only approximate. The constantsdi(cid:11)er for each limb and are only valid within a certain movement amplitudein 2D space, therefore the extrapolation of the data outside that range andinto 3D space has no validated experimental basis. Nonetheless, Fitts’ Lawprovides a reasonable and easily computed basis for approximating movementdurations.While it may seem odd at (cid:12)rst to have attacked both Natural Languageinterpretation and timing constructs as part of the same research, Esakov’swork foreshadows our more recent work on language and animation by fo-cusing on the fact that the same instruction, given to agents with di(cid:11)erentabilities, will be carried out in di(cid:11)erent ways. Language tells an agent whathe or she should attempt to do: how he or she does it depends on them.6.2 Language Terms for Motion and Space2The next piece of work that was done on driving task animation throughNatural Language commands was Jugal Kalita’s work on how Natural Lan-guage conveys desired motion and spatial aspects of an agent’s behavior. InEnglish, the primary burden falls on verbs and their modi(cid:12)ers. Kalita’s workshowed how verbs and their modi(cid:12)ers can be seen as conveying spatial andkinematic constraints on behavior, thereby enabling a computer to create ananimated simulation of a task speci(cid:12)ed in a Natural Language utterance. Thiswork is described at greater length in [Kal90, KB90, KB91].6.2.1 Simple CommandsTo understand Kalita’s contribution, consider the following commands:(cid:15) Put the block on the table.(cid:15) Turn the switch to position 6.(cid:15) Rol l the bal l across the table.Each of these commands speci(cid:12)es a task requested of an agent. Perform-ing the task, requires inter alia that the agent understand and integrate themeanings of the verbs (put, turn, open, rol l) and the prepositions (on, to,across). This requires understanding the signi(cid:12)cant geometric, kinematic ordynamic features of the actions they denote.In Kalita’ approach to physically based semantics, a motion verb denoteswhat may be called a core or kernel action(s). This kernel representationis then used with ob ject knowledge and general knowledge about actions toobtain semantic representations and subsequent task performance plans whichare speci(cid:12)c to a context { for example,2Jugal Kalita.6.2. LANGUAGE TERMS FOR MOTION AND SPACE215(cid:15) The core meaning of the verb put (as in Put the block on the table)establishes a geometric constraint that the (cid:12)rst ob ject (here, the block)remains geometrically constrained to (or, brought in contact with andsupported by) a desired position on the second ob ject (here, the table).(cid:15) The core meaning of the verb push (as in Push the block against thecone) involves applying a force on the manipulated ob ject (here, theblock) through the instrument ob ject (here, the hand). The preposi-tional phrase speci(cid:12)es the destination of the intended motion.(cid:15) The verb rol l (as in Rol l the bal l across the table) involves two relatedmotions occurring simultaneously|one rotational about some axis ofthe ob ject, and the other translational, caused by the (cid:12)rst motion. Therotational motion is repeated an arbitrary number of times.6.2.2 Representational FormalismGeometric relations and geometric constraintsThe meanings of locative prepositions are represented using a template calleda geometric-relation. A simple geometric relation is a frame-slot structure:geometric-relation:spatial-type:source-constraint-space:destination-constraint-space:selectional-restrictions:Spatial-type refers to the type of the geometric relation speci(cid:12)ed. Its valuesmay be positional or orientational. The two slots called source-constraint-space and destination-constraint-space refer to one or more ob jects, or partsor features thereof, which need to be related. For example, the commandPut the cup on the table requires one to bring the bottom surface of the cupinto contact with the top surface of the table. The command Put the bal lon the table requires bringing an arbitrary point on the surface of the ballin contact with the surface of the table top. Since the items being relatedmay be arbitrary geometric entities (i.e., points, surfaces, volumes, etc.), wecall them spaces. The (cid:12)rst space is called the source-constraint space andthe second, the destination-constraint space. The slot selectional-restrictionsrefers to conditions (static, dynamic, global or ob ject-speci(cid:12)c) that need tobe satis(cid:12)ed before the constraint can be executed.More complex geometric relations require two or more geometric relationsto be satis(cid:12)ed simultaneously:geometric-relation:fg-uniong-relation-1g-relation-2(cid:1) (cid:1) (cid:1)g-relation-n g216CHAPTER 6. TASK-LEVEL SPECIFICATIONSwhere g-relation-i is simple or complex.Geometric relations are also used in the speci(cid:12)cation of geometric con-straints, which are geometric goals to be satis(cid:12)ed:Geometric-constraint: execution-type:geometric-relation:Geometric constraints are distinguished by their execution-type slot, whichcan take one of four values: achieve, break, maintain or modify.The frame used for specifying kinematic aspects of motion is the following:KinematicsKinematics: motion-type:source:destination:path-geometry:velocity:axis:Motions are mainly of two types: translational and rotational. In order todescribe a translational motion, we need to specify the source of the motion,its destination, the tra jectory of its path (path-geometry), and the velocityof the motion. In the case of rotational motion, path-geometry is circular,and velocity, if speci(cid:12)ed, is angular. Rotational motion requires an axis ofrotation.If not speci(cid:12)ed, it is inferred by consulting geometric knowledgeabout the ob ject concerned.Kernel actionsThe central part of an action consists of one or more components: dynamics,kinematics and geometric-constraints|along with control structures statingits other features. The control structures used in the examples that followare: repeat-arbitrary-times and concurrent. The keyword concurrent is speci-(cid:12)ed when two or more components, be they kinematic, dynamic or geometricconstraints, need to be satis(cid:12)ed or achieved at the same time. The keywordrepeat-arbitrary-times provides a means for specifying the frequentation prop-erty of certain verbs where certain sub-action(s) are repeated several times.The verbs’ semantic representation need not specify how many times the ac-tion or sub-action may need to be repeated. However, since every actionis presumed to end, the number of repetitions of an action will have to becomputed from simulation (based on tests for some suitable termination con-ditions), or by inference unless speci(cid:12)ed linguistically as in Shake the blockabout (cid:12)fty times.6.2. LANGUAGE TERMS FOR MOTION AND SPACE2176.2.3 Sample Verb and Preposition Speci(cid:12)cationsMany of the features of Kalita’s representation formalism can be seen in hisrepresentation of the verbs \roll" and \open" and the prepositions \in" and\across". Others can be seen in the worked example in Section 6.2.4.A kinematic verb:rol lThe verb rol l refers to two motions occurring concurrently: a rotational motionabout the longitudinal axis of the ob ject and a translational motion of theob ject along an arbitrary path. The rotational motion is repeated an arbitrarynumber of times. The verb rol l is thus speci(cid:12)ed as:roll (l-agent, l-ob ject, path-relation)agent: l-agentob ject: l-ob jectkernel-action:concurrentff f kinematic:motion-type:rotationalaxis:longitudinal-axis-of (l-ob ject)ggrepeat-arbitrary-timesf kinematic:motion-type:translationalpath:path-relation g gselectional restrictions:has-circular-contour (l-ob ject,longitudinal-axis-of (l-ob ject))A verb that removes constraints:openOne sense of open is to move (as a door) from closed position. The meaningis de(cid:12)ned with respect to a speci(cid:12)c position of a speci(cid:12)c ob ject. The closedposition of the ob ject can be viewed as a constraint on its position or orien-tation. Thus, this sense of open involves an underlying action that undoesan existing constraint. The ob ject under consideration is required to have atleast two parts: a solid 2D part (the cover) and an un(cid:12)lled 2D part de(cid:12)nedby some kind of frame (the hole). The meaning must capture two things: (1)that at the start of the action, the ob ject’s cover must occupy the total spaceavailable in ob ject’s hole in the constrained position, and (2) that the resultof the action is to remove the constraint that ob ject’s cover and its hole arein one coincident plane. This is ful(cid:12)lled by requiring that the two sub-ob jects(the hole and the cover) are of the same shape and size.The de(cid:12)nition for open is:open (Ag, Ob j)agent: Agob ject: Ob jkernel-action:geometric-constraint:218CHAPTER 6. TASK-LEVEL SPECIFICATIONSexecution-type:breakspatial-type:positionalgeometric-relation: source-constraint-space: Ob j (cid:15) holedestination-constraint-space: Ob j (cid:15) coverselectional-restrictions: contains-part (Ob j, hole)contains-part (Ob j, cover)area-of (Ob j (cid:15) cover) = area-of (Ob j (cid:15) hole)shape-of (Ob j (cid:15) cover) = shape-of (Ob j (cid:15) hole)A locative preposition:inThe sense of in captured here is within the bounds of, contained in or includedwithin. According to Herskovits [Her86], this use type for in is spatial entityin a container. This meaning of in is speci(cid:12)ed asin (X,Y)  (cid:0)geometric-relation:spatial-type:positionalsource-constraint-space:volume-of (X)destination-constraint-space:interior-of (Y)selectional-restrictions:or (container-p (Y), container-p (any-of (sub-parts-of (Y))))size-of (X) (cid:20) size-of (Y)normally-oriented (Y)A container is an ob ject which can hold one or more ob jects such that the ob-ject is \surrounded by" the volume de(cid:12)ned by the boundaries of the container.It is a concept which is di(cid:14)cult to de(cid:12)ne clearly, although heuristics can bedevised to recognize whether an ob ject is a container. For our purposes, ifan ob ject or any of its part(s) can work as container(s), it will be so labeledin the function slot of its representation. The second selectional restrictionis due to Cooper [Coo68]. The third restriction is due to Herskovits, whoexplains its necessity by stating that the sentence The bread is in the bow l ispragmatically unacceptable if the bowl is upside down and covers the breadunder it [Her86].A path preposition:acrossPath is a part of kinematic speci(cid:12)cation of a motion or an action. A com-plete de(cid:12)nition of path requires specifying its source, destination and pathgeometry, which Kalita does, using a structure called a path-speci(cid:12)cation:path-speci(cid:12)cation:source:destination:path-geometry:6.2. LANGUAGE TERMS FOR MOTION AND SPACE219Across is one of several path prepositions in English. Others include from,to, around, round and along. Across has two types of meanings|dynamicand static (locative) meaning. The dynamic meaning implies a journey acrossan ob ject, whereas the static meaning implies a location between two lines(edges) perpendicular to them and touching, and (possibly) extending beyondthem. The dynamic sense of across is seen in:(cid:15) Roll/Slide/Move the block/ball across the board.This dynamic sense of across speci(cid:12)es all three components required for pathsource:any-of (exterior-edges-of (Y, parallel-to (longitudinal-axis (Y))))destination:any-of (exterior-edges-of (Y, parallel-to (longitudinal-axis (Y))))speci(cid:12)cation.across (X, Y)  (cid:0) path-speci(cid:12)cation:path-geometry: straight-lineselectional-restrictions:destination 6= sourcehas-axis (X, longitudinal)(cid:14)angle-between (path-geometry, longitudinal-axis (Y), 90)length (Y) (cid:21) width (Y)length (Y)dimension-of (X,along-direction (longitudinal-axis (Y)))>The longitudinal axis of an ob ject is the axis along which the length of anob ject is measured. There are a number of selectional restrictions imposed onthe ob jects X and Y also. For example, the reason for the fourth selectionalrestriction can be gauged from the two phrases: across the road and along theroad.6.2.4 Processing a sentenceThe sentence Put the block on the table can be used to show how Kalita’ssystem obtains a meaning for a whole sentence from the meanings of its parts,i.e., the lexical entries of its constituent words.The lexical entry for put speci(cid:12)es the achievement of a geometric relation-ship between an ob ject and a location speci(cid:12)ed by a prepositional phrase. Themeaning of the verb is speci(cid:12)ed in terms of a yet-unspeci(cid:12)ed geometric rela-tion between two ob jects. The preposition on along with the ob jects involvedleads to the sense that deals with support.A bottom-up parser [FW83] returns the logical meaning representationas (put you block-1 (on block-1 table-1 )).In this representation, the verbput takes three arguments: a sub ject, an ob ject and the representation fora locative expression. Entities block-1 and table-1 are ob jects in the worlddetermined to be the referents of the noun phrases. The logical representationhas you as the value of the sub ject since the sentence is imperative.Now, to obtain the intermediate meaning representation, the argumentsof put in the logical representation are matched with the arguments in thefollowing lexical entry for put:220CHAPTER 6. TASK-LEVEL SPECIFICATIONSput (l-agent, l-ob ject, l-locative)  (cid:0)agent: l-agentob ject: l-ob jectkernel-actions:geometric-constraint:execution-type:achievegeometric-relation: l-locativeThis lexical entry has three arguments. After matching, l-agent has the valueyou, l-object has the value block-1, and l-locative has the value (on block-1table-1). The value of the geometric-relation slot (of the kernel-actions slot inthe representation) is (cid:12)lled in by the semantic representation for the l-locativeargument which is created from the meaning of \on the table", using thefollowing de(cid:12)nition of \on":on (X,Y)  (cid:0)geometric-relation:spatial-type:positionalsource-constraint-space:any-of (self-supporting-spaces-of (X))destination-constraint-space: any-of (supporter-surfaces-of (Y))selectional-restrictions:horizontal-p (destination-constraint-space)equal (direction-of (normal-todestination-constraint-space), \global-up")free-p (destination-constraint-space)As a result, the intermediate meaning representation of \put the block onthe table" is:agent:youob ject:block-1kernel-actions:geometric-constraint:execution-type:achievegeometric-relation:spatial-type:positionalsource-constraint-space:any-of(self-supporting-spaces-of (block-1))destination-constraint-space: any-of(supporting-surfaces-of (table-1))selectional-restrictions:horizontal-p (destination-constraint-space)equal (direction-of (normal-todestination-constraint-space), \global-up")free-p (destination-constraint-space)In order to execute the action dictated by this sentence, the program looksat the knowledge stored about the block to (cid:12)nd a part of the block on which6.2. LANGUAGE TERMS FOR MOTION AND SPACE221it can support itself. It observes that it can be supported on any one of itsfaces and no face is more salient than any other. A cube (the shape of theblock) has six faces and one is chosen randomly as the support area. Next,the program consults the knowledge stored about the table and searches fora part or feature of the desk which can be used to support other ob jects. Itgathers that its function is to support \small" ob jects on its top. This topsurface is also horizontal. As a result, (cid:12)nally, the system concludes that oneof the sides of the cube has to be brought in contact with the top of the table.The (cid:12)nal meaning for the sentence obtained isagent: youob ject: block-1kernel-actions:geometric-constraint:execution-type:achievegeometric-relation:spatial-type:positionalsource-constraint-space:block-1(cid:15)side-2destination-constraint-space: table-1(cid:15)top-1block-1(cid:15)side-2 represents a speci(cid:12)c face of a speci(cid:12)c block. table-1(cid:15)top-1 rep-resents the top surface of a speci(cid:12)c table. This (cid:12)nal representation is thensent to a planner [JKBC91] which produces a plan for performing the task byan animated agent in a given workspace. The plan is taken up by a simula-tor [BWKE91] which establishes connection with Jack and then produces ananimation:The block is initially sitting on top of a closed box. The agentreaches for it with his right hand, grasps it, moves it to a pointnear the top of a table to his left, places it on the table, and moveshis hand back.As with Esakov’s work, there were still unfortunate capability gaps in thesimulator available to Kalita.In particular, the lack of a (cid:13)exible torso,unchecked collisions with the environment, and no balance constraints led tosome painful-looking postures and ob ject tra jectories which passed throughobstacles.6.2.5 SummaryThis section has discussed the representation of meanings of some verbs andprepositions, emphasizing the importance of geometric information such asaxes of ob jects, location of ob jects, distance or angle between ob jects, path ofob ject motion, physical contact between ob jects, etc., in the meaning repre-sentation of prepositions. Elsewhere it is shown that such geometric consid-erations are important for not only representing verbs and prepositions, butalso adverbs [KB90].222CHAPTER 6. TASK-LEVEL SPECIFICATIONSIn the work described here, the operational meanings of action verbs andtheir modi(cid:12)ers have been represented in terms of components pertaining toconstraints and kinematic/dynamic characterization. For additional examplesof decomposition see [Kal90].6.3 Task-Level Simulation3The third experiment tested the feasibility of using what might be viewed aslow-level task primitives to create task animations [Lev91]. If successful, thiswould have two advantages:(cid:15) Since we viewed some kind of low-level task primitives as being theoutput speci(cid:12)cation language of any language processing stages, it wouldallow us to design and test a set of primitives in parallel with the othersystem components.(cid:15) This kind of lower-level speci(cid:12)cation language might itself be usable byan engineer to generate animations in terms of task-level actions ratherthan having to specify particular body movements.To illustrate the latter contrast, consider a scene with an animated agent,a table, and a cup on a shelf next to the table. The animator{engineer wantsto create an animation of the agent moving the cup from the shelf to the table.A task-level speci(cid:12)cation could enable the animator{engineer to produce thedesired behavior, using a set ofspeci(cid:12)cations. For example, thetask-actionsequencegrasp-action(hand cup)position-action(cup table-top)could be used to generate an animation of the agent’s hand grasping the cup,followed by a positioning of the cup on the top of the table.As a test environment we used an expanded version of some written in-structions to remove a Fuel Control Valve (FCV) from an imaginary aircraftfuselage (Figure 6.1).Fuel Control Valve Removal Instructions:1. With right hand, remove socket wrench from tool belt, move to front ofbody. With left hand, reach to tool belt pocket, remove 5/8 inch socket,move to wrench, engage. Adjust ratchet for removal.2. Move wrench to left hand bottom hole, apply pressure to turn in aloosening motion, repeat approximately 7 times to loosen threaded bolt.3. Move wrench away from bolt, with left hand reach to bolt and removebolt and washer from assembly, move left hand to belt pouch, place boltand washer in pouch.3Libby Levison.6.3. TASK-LEVEL SIMULATION223Figure 6.1: A Frame from the Fuel Control Valve Removal Task. The FCV isthe Cylindrical Ob ject Mounted to the Flat Plate.4. Move wrench to bottom right hand bolt, apply pressure to turn in aloosening motion, repeat approximately 7 times to loosen threaded bolt.5. Repeat operation 3.6. Move wrench to top bolt, apply pressure to turn in a loosening motion,repeat approximately 6 times to loosen threaded bolt. Move left handto grasp assembly, loosen the bolt the (cid:12)nal turn. Move wrench to toolbelt, release. With right hand reach to bolt, remove bolt and washer,place in pouch. Return right hand to assembly, with both hands moveFuel Control Valve to movable cart and release.6.3.1 Programming EnvironmentThe work area, tools and parts for the scene were modeled with Jack. Just asthe engineer who currently writes the instruction manuals has knowledge ofthe task and knows, for example, that a Phillips head screwdriver is required,it is assumed that the engineer-animator will have the knowledge required tolay out the scene of the animation. It is also assumed that a skilled engineer isalready trained in analyzing tasks and developing instruction sets for the do-224CHAPTER 6. TASK-LEVEL SPECIFICATIONSmain. This pro ject simply provides a di(cid:11)erent medium in which the engineercan explain the task.The task simulation is based on, a symbolic process simulator [EB90,YapsBWKE91].provides animation-directives which access Jack’s behaviors.YapsThese animation-directives are not only ordered and sequenced viaYaps’temporal and conditional relationships [KKB88], but can also be composedto produce parameterized simulation procedures. These procedures, calledtask-actions, are de(cid:12)ned for a number of parameters (agent, ob ject, loca-tion, etc.). The same task-action can thus be used at various times withdi(cid:11)erent parameters to create distinct animation segments. The possibility ofde(cid:12)ning and reusing these procedures simpli(cid:12)es the animation programmingproblem for the engineer. By extending these procedural compositions, high-level procedures could be generated so that the mapping from the instructionsto these procedures would be straightforward.KB[Esa90] is a frame-based, ob ject-oriented knowledge system whichestablishes symbolic references to Jack’s geometric data. While Jack main-tains and manipulates the geometric model of the world,maintains theKBsymbolic information.uses’s symbolic representation to manipu-YapsKBlate the geometric model. (These symbolicrepresentations are passedKBto thetask-actions as parameters.) This freesfrom \knowing"YapsYapsthe speci(cid:12)c world coordinates of an ob ject or the ob ject’s exact geometricrepresentation. For instance, if Jack contains a model of a cup,wouldKBhave an entry which identi(cid:12)ed cup as that particular Jack entity.has noYapsknowledge of the ob ject’s location;’s mapping from symbolic to geometricKBrepresentation will resolve any ambiguity. Thus the animator need not talkabout the-cup-on-the-table-at-world-coordinates-(x,y,z), but can reference thesymbolic entity, cup. Should the cup move during the course of the action,KBresolves the problem of the cup’s exact location.6.3.2 Task-actionsAt the time of this research,provided only three low-level animation-Yapsdirectives with which to access Jack behaviors. These are generate-motion,create-constraint and delete-constraint. Generate-motion causes an ob ject(not necessarily animate) to move from its current location to another. (Nopath planning was performed in the Jack version of the time, andYapshandled frame-to-frame timing directly as described in Section 6.1.) Create-constraint establishes a physical link between two (not necessarily adjacent)ob jects. If two ob jects are linked together and one of the ob jects is moved,the second ob ject moves along with it. The physical constraint (relation) be-tween the ob jects is maintained. Create-constraint can be further speci(cid:12)ed touse positional and/or orientational alignments. Delete-constraint removes thespeci(cid:12)ed constraint between two ob jects.Yapsprovides a mechanism for building animation templates by combin-ing or composing the above animation-directives. Using di(cid:11)erent combina-tions of generate-motion, create-constraint, and delete-constraint, and vary-6.3. TASK-LEVEL SIMULATION225ing the agents and the ob jects of these animation-directives as well as theirtemporal and causal relations, it is possible to build a set of task-actions.Task-actions can themselves be composed into more complex task-actions.As the procedures acquire more speci(cid:12)cation, the task-actions approach task-level descriptions. It is important to note, however, that task-actions simplyde(cid:12)ne templates; an animation is realized by instantiating the task-actions,supplying parameters as well as timing constraints and other conditions. Thecomposability of the task-actions allows for the de(cid:12)nition of some abstractand high-level concepts. It is these high-level animation descriptions whichwill allow the engineer to program an animation at the task-level.6.3.3 Motivating Some Task-ActionsThe (cid:12)rst templates to be de(cid:12)ned were simply encapsulations of the Jackanimation-directives:(agent ob ject),(agent ob-reach-actionhold-actionject) and(ob ject) { were just generate-motion, create-free-ob ject-actionconstraint and delete-constraint, respectively.(Although the names chosenfor the task-actions do make some attempt to elicit their de(cid:12)nition, there wasno attempt to come up with de(cid:12)nitive de(cid:12)nitions of these actions in this seg-ment of the research pro ject.) In the following, the use of agent and ob jectis simply for readability; for example, acan be applied betweenhold-actiontwo ob jects (e.g.,(wrench-head 5-8th-socket)).hold-actionConsider trying to describe the actions inherent in the example:Move the cup to the tableassuming that the agent is not currently holding the cup. The agent must(cid:12)rst hold the cup before he can move it. How is this animation speci(cid:12)ed?Explicitly stating the sequence of actions:reach-action(agent cup)hold-action(agent cup)to cause the agent to reach his hand to the location of the cup and to constrainhis hand to the cup seems awkward. Composing two task-actions allows a newtask-actionto be de(cid:12)ned:grasp-action(deftemplate(agent ob ject)grasp-actionreach-action(agent ob ject)hold-action(agent ob ject)).(This is the actualde(cid:12)nition.is thecommand toYapsDeftemplateYapsde(cid:12)ne a new task-action template.)is a sequence of instanti-Grasp-actionations of two primitive task-actions.Now that the agent can grasp the cup, how can he move the cup? A secondaction,, is de(cid:12)ned to relocate the cup to a new location andposition-actionconstrain it there:226CHAPTER 6. TASK-LEVEL SPECIFICATIONS(deftemplate(ob ject1 location)position-actionreach-action(ob ject1 location)hold-action(ob ject1 location)).If a previous action had left an ob ject (the cup) in the agent’s hand, thistask-action could be used to move the ob ject to a new location (position-actioncup table). (In this instruction set, the only use of the instruction\move something that is already being held" required that the ob ject beconstrained to the new location. This is the justi(cid:12)cation of thehold-actionin this de(cid:12)nition.) Note here that location could be the location of ob ject2.Thus, to animate the instruction:Move the cup to the tablethe animation-script could be:grasp-action(agent-right-hand cup)position-action ()cup table-top.It is still necessary to specify a list of commands, since no high-level task-action has been de(cid:12)ned for move, and therefore the action must be describedin increments. Howevercould be de(cid:12)ned as:move-action(deftemplate(agent ob ject1 location)move-actiongrasp-action(agent ob ject1)position-action(ob ject1 location)).In other words, grasp (reach to and hold) ob ject1, and position (move to andconstrain) ob ject1 at location (where location might be the location of someob ject2). In the Move the cup example, the instantiation required to achievethe desired animation would be:move-action(agent-right-hand cup table-top).This conciseness is one bene(cid:12)t of task-action composition.Once the cup is actually on the table, it can be \un-grasped" by using:free-ob ject-action(cup)which breaks the constraint between the hand and the cup. If the hand islater moved, the cup will no longer move with it.The (cid:12)nal animation script for Move the cup to the table becomes:move-action(agent-right-hand cup table-top)free-ob ject-action(cup).6.3.4 Domain-speci(cid:12)c task-actionsThe Move the cup to the table example motivated a few fundamental task-action de(cid:12)nitions. Some of these are actions common to many instructional6.3. TASK-LEVEL SIMULATION227tasks and milieus; this set of task-actions is also usable in the instruction setdescribing the FCV removal. However, it was also necessary to return to theinstruction set and developde(cid:12)nitions for actions speci(cid:12)c to the domainYapsin question. These task-actions can be either primitive (seeturn-actionbelow) or compositional (see). The (cid:12)rst new task-action,ratchet-actionattach-action, is de(cid:12)ned as:(deftemplate(agent ob ject1 ob ject2)attach-actionmove-action(agent ob ject1 ob ject2)hold-action(ob ject1 ob ject2)).This allows the agent to grasp ob ject1, move it to the location of ob ject2, andestablish a constraint between ob ject1 and ob ject2. The expansion of thistask-action is the command string:reach-action, hold-action, reach-action, hold-action.Attach-actioncould have been equivalently de(cid:12)ned as:(deftemplate(agent ob ject1 ob ject2)attach-actiongrasp-action(agent ob ject1)position-action(ob ject1 ob ject2))which would expand to exactly the same Jack animation-directive commandstring as above. The task-action de(cid:12)nitions are associative; this provides(cid:13)exibility and power to the system, and increases the feasibility of de(cid:12)ning aminimal set of task-actions to be used throughout the domain.The FCV removal instructions also require:(ob ject degrees).turn-actionTurn-actioncauses the ob ject to rotate by the speci(cid:12)ed number of degrees.The geometric de(cid:12)nition of the ob ject includes information on its DOFs; forexample, around which axis a bolt will be allowed to rotate. At the time thatthis research was done, the system did not have a feedback tool to monitorJack entities; instead of testing for an ending condition on an action (a boltbeing free of its hole), actions had to be speci(cid:12)ed iteratively (the number oftimes to turn a bolt).is actually a support routine, used in theTurn-action(cid:12)nal task-action needed to animate the FCV instructions:ratchet-action.This is de(cid:12)ned as:(deftemplate(ob ject degrees iterations)ratchet-actionturn-action(ob ject degrees)turn-action(ob ject (cid:0)degrees)ratchet-action(ob ject degrees iterations(cid:0)1)).Ratchet-actionis used to animate of a socket wrench ratcheting back and4forth.4Having to explicitly state a number of degrees is not an elegant programming solution;it would have been preferable to take advantage of’s collision detection algorithms toJackdetermine the range of the ratchet movement. Processing considerations at the time thework was done required this rather rough implementation.228CHAPTER 6. TASK-LEVEL SPECIFICATIONSThe complete set of task-actions is listed below. With this set of onlynine task-actions, it was possible to program the entire animation script fromthe natural language instructions (see Table 6.2 for an excerpt of the (cid:12)nalanimation script).reach-actionagent ob ject()hold-actionagent ob ject()free-ob ject-actionob ject()grasp-actionagent ob ject()move-actionagent ob ject location()attach-actionagent ob ject1 ob ject2()position-actionob ject1 ob ject2()turn-actionob ject degrees()ratchet-actionob ject degrees iterations()(cid:15)(cid:15)(cid:15)(cid:15)(cid:15)(cid:15)(cid:15)(cid:15)(cid:15)6.3.5 IssuesWhere Does Task-Action Decomposition Stop?There is an interesting question as to whether, in de(cid:12)ning task-actions, oneneeds to be concerned with variations that arise from di(cid:11)erences in agentsand their abilities.Because our work is embedded in Jack, variations in agent ability at theanimation speci(cid:12)cation level is not a concern. As long as the animation iswithin the agent’s capabilities (and thus the animation is \solvable"), sub-stituting di(cid:11)erent agents gives di(cid:11)erent valuations of the tasks. By testingdi(cid:11)erent agents with varying abilities, one can analyze the task requirementsand gather information on human factors issues. Similarly, it is possible tovary workplace geometry, tools, and agent placement.Note the comparison here between innate and planned action. In reachingto grab a cup, we do not think about how to control the muscles in the forearm;we do, however, consider the goal of getting our hand to the same location asthe cup. This distinction between cognizant motion and action is internal inthis animation; Jack manages the motor skills. The same distinction is foundin the level of detail of the instructions. One does not tell someone:Extend your hand to the cup by rotating your shoulder joint 40(cid:14)(cid:14)while straightening your elbow joint 82degrees. Constrain yourhand to the cup by contracting (cid:12)ngers.: : :Rather, we give them the goal to achieve and allow that goal to lend informa-tion as to how to accomplish the instruction. The hierarchy of the task-actionscaptures some of this knowledge.The task-actions have been de(cid:12)ned in such a way that they are not con-cerned with the abilities of a speci(cid:12)c agent, but rather allow for interpretation;;;6.3. TASK-LEVEL SIMULATION229Table 6.2: Animation Script Excerpt.;;; No. 1;;;With right hand, remove socket wrench from tool belt,;;;move to front of body. With left hand, reach to tool belt;;;pocket, remove 5/8" socket, move to wrench, engage.;;;Adjust ratchet for removal.; with the right hand, grasp the wrench from the tool belt,; and move it to site-front-body(instantiate move-action(fred-rh wrench-handle fred-front-body-site planar):instancename "r0-wrench-to-front":time-constraints ’((start now)(duration(eval(+(fitts fred-rh wrench-handle)(fitts wrench-handlefred-front-body-site))))); with the left hand, attach socket to wrench handle.; an attach entails, reaching for the socket, grasping; it and moving it to the wrench head.; if successful, free the left hand from the socket.(instantiate attach-action(fred-lh 5-8th-socket wrench-headattach-socket-time planar oriented):instancename "r5-attach-socket":time-constraints ’((start (end "r0-wrench-to-front"))(duration (eval(+ (fitts fred-lh 5-8th-socket)(fitts fred-left-pocketfred-front-body-site)attach-socket-time)))):on-success ’(progn(free-object-action fred-lh)(free-object-action 5-8th-socket)(hold-action wrench-head 5-8th-socket:orientation-type ’("orientation")))230CHAPTER 6. TASK-LEVEL SPECIFICATIONSbased on each agent’s capabilities. Not only does this allow the same anima-tion script to be used for di(cid:11)erent agents, generating di(cid:11)erent analyses, but italso means that the de(cid:12)nitions of the task-actions decomposition stops at thelevel of innate action. There is no need to have multiple task-action de(cid:12)nitionsfor various physical attributes; Jack handles this issue for us.Instruction TranslationWe noted earlier that one advantage of alevel of speci(cid:12)cationtask-actionwas that it might allow an engineer/animator to animate tasks directly. Interms of the above task-actions, moving the cup to the table (noted in theintroduction) could be animated by issuing either of two command sequences:grasp-action(agent-right-hand cup)position-action(cup table-top)free-ob ject-action(cup)or:move-action(agent-right-hand cup table-top)free-ob ject-action(cup).In both cases, the engineer has decided to release the constraint between theagent and the cup as soon as the cup is on the table-top. The engineer hasalso described the required animation at the task-level.Sequencing Sub-tasksYapsis a simultaneous language; that is, all task-action instantiations areresolved concurrently. To sequence the actions and force them to occur in aspeci(cid:12)c order, the engineer/animator must use the timing-constraints optionprovided by. This construct allows the user to specify starting, endingYapsand duration conditions for the instantiation of each action. It is possible toachieve the ordering needed to create a sequential animation by predicatingthe starting condition of instruction-2 on the ending condition of instruction-1; but a task-action template, which is de(cid:12)ned as a series of other task-actions,has the sequencing automatically built in via the instantiation process. If thiswere not the case, de(cid:12)ning, for example, would be impossiblegrasp-actionbecause achieving and completing thebefore starting thereach-actionhold-actioncould not be guaranteed.The actions do not need to be performed discretely. OthertimingYapsconstructs allow the actions to be overlapped and delayed by specifying (start(after 5 min)) or (start now), for example [KKB88]. Nor is de(cid:12)ning a discretelinear order on the sub-tasks the only possibility. The simultaneous nature ofYapsis used to animate actions (such as moving an ob ject with both hands)by simultaneously animating:move-action(agent-left-hand box)move-action(agent-right-hand box).6.4. A MODEL FOR INSTRUCTION UNDERSTANDING231Thetiming constraints provide a powerful mechanism for specifyingYapsthe relationships among the task-actions in the animation. Timing is one ofthe most critical issues involved in generating realistic animations; the powerthatprovides in resolving timing issues greatly enhances the potentialYapsof the Jack animation system.Task DurationThetiming constraints provide a powerful mechanism for specifying theYapsinter-relationships among the task-actions in the animation script. Timing isone of the most critical issues involved in generating realistic animations.We have already noted that it is not su(cid:14)cient to simply list all the actions;they must be times, sequenced and connected temporally. As in Esakov’swork, adaptations of Fitts’ Law were used to determine mimimum actiontimes. Fitts’ Law was used to calculate the duration of allreach-actioninstantiations. Thus, time requirements were cumulative (i.e., the sum ofthe sub-task-action times).uses a small default constantCreate-constrainttime to estimate sub-task duration. Although Fitts’ Law only approximatesthe action times in this domain and must be further scaled by a motivationfactor, it does give reasonable estimates. Relative to one another, the sub-task times make sense. Although the length of eachmight nottask-actionbe correct, the animation does appear to be temporally coherent.6.3.6 SummaryRecent work in de(cid:12)ning animation behaviors reviewed earlier in this bookgreatly expands the set of animation directives available in Jack. In our cur-rent work, we will investigate using the new animation behaviors to scriptanimations. Since animation directives form the semantical basis for our ac-tion de(cid:12)nitions, a more powerful set of animation directives provides us witha richer language with which to work. As it becomes easier to de(cid:12)ne newtask-actions, the animator will spend less time coordinating sub-actions.Finally, this new vocabulary will allow us to express tasks (or de(cid:12)ne task-actions) which di(cid:11)er from the earlier work in their semantic content. Our (cid:12)rstattempt at rescripting the instruction set resulted in a more realistic anima-tion, in that the new behaviors allowed us to include such low-level actionsaswhen the animated agent was reachingtake step to maintain balancebeyond his comfort range. We need to compare the expressive powers of theprevious animation directives with the enhanced set of animation behaviors.6.4 A Model for Instruction UnderstandingThe three experiments described in the previous sections were all concernedwith the operational semantics of single-clause commands. But the range ofBarbara Di Eugenio, Michael White, Breck Baldwin, Chris Geib, Libby Levison,55Michael Moore.232CHAPTER 6. TASK-LEVEL SPECIFICATIONStasks that can be communicated to an agent with such commands is very lim-ited { the less expertise and experience on an agent’s part, the more he needsto be told. A telling example of this is given in [Pri81]. Here, Prince comparesa recipe for stu(cid:11)ed roast pig given in a nineteenth century French cookbookwith that given in Rombauer’s contemporary The Joy of Cooking. The formersays, essentially, \Roast pig. Stu(cid:11) with farce anglaise." Rombauer’s instruc-tions go on for two pages: she assumes very little culinary experience withpigs on the part of today’s men and women.Multi-clause commands are very common in maintenance and assembly in-structions, such as the following examples from Air Force maintenance manualT.O. 1F-16C-2-94JG-50-2:\With door opened, adjust switch until roller contacts cam andcontinuity is indicated at pins A and B. Verify positive switchcontact by tightening bottom nut one additional turn." (p. 5-24)\Hold drum timing pin depressed and position entrance unit ondrum. Install three washers and three bolts, release drum timingpin, and torque bolts to 60-80 inch-pounds." (p. 6-14)Now just as multi-clause texts are commonly organized into paragraphs,multi-clause instructions are commonly organized into steps.In fact, theabove multi-clause commands are actually single steps from longer, multi-step instructions. While there are no (cid:12)rm guidelines as to what a singleinstruction step should encompass, there is a strong tendency at least forsteps to be organized around small coherent sub-tasks (such as adjusting aswitch or installing a component, as in the above examples). A typical stepmay specify several actions that need to be performed together to accomplish asingle subtask, or several aspects of a single complex action (e.g. its purpose,manner, things to watch out for, appropriate termination conditions, etc.).The agent must develop some degree of understanding of the whole step beforestarting to act.In our current work on instruction understanding, we add to this sub-tasksense of step, the sense that a step speci(cid:12)es behavior that the agent mustattend to continuously: while carrying out a step, the agent’s attention is(cid:12)xed on the task at hand. Communication with the instructor is not alloweduntil completion (or failure) of the current step. Because of this, a step de(cid:12)nesthe extent of the instructions that must be processed before the agent beginsto act on them. (With some re(cid:13)ection on one’s own confrontations with newinstructions, it is easy to recall situations where one has tried to understandtoo much or to act on too little understanding. It is not always obvious whenone should begin to act.)While our focus is on multi-clause instructions, it turns out that manyof their important features can be demonstrated simply with two-clause in-structions. (As in many things, the biggest leap is from one to two.) Thetwo-clause example we will use here to describe our framework for instructionunderstanding and animation is:6.4. A MODEL FOR INSTRUCTION UNDERSTANDING233\Go into the kitchen to get me the co(cid:11)ee urn."This example will be used to illustrate, among other things:(cid:15) expectations raised by instructions;(cid:15) the need for incremental generation of sub-goals (plan expansion) inorder to act in accordance with instructions;(cid:15) the need to accommodate the agent’s behavior in carrying out actions,to the ob jects being acted upon; and(cid:15) the need to develop plans at more than one level.Figure 6.2 shows a schematic diagram of the AnimNL (ANIMation fromNatural Language) architecture. Before going through the example, we wantto call attention to the system’s overall structure { in particular, to the factthat it consists of two relatively independent sets of processes: one set ofwhich produces commitments to act for a particular purpose, what we callanimated task actions { e.g.(cid:15) goto(door1, open(door1)) { \go to door1 for the purpose of opening it"(cid:15) grasp(urn1, carry(urn1)) { \grasp urn1 for the purpose of carrying it"and the other set of which (cid:12)gures out how the agent should move in orderto act for that purpose. In this framework, instructions lead to initial com-mitments to act, and actions once embarked upon allow further commitmentsto be made and acted upon.(While our discussion here will be in termsof single-agent procedures, it can be extended to multi-agent procedures byadding communicative and coordinating actions. As shown in earlier chapters,both Jack and its behavioral simulator can support the activity of multipleagents. However, extending the upper set of processes to delineate the com-munication and coordination required of multiple agents cooperating on a taskrequires solution of many problems currently under investigation by membersof the AI planning community.We now begin by giving AnimNL the instruction step:\Go into the kitchen to get me the co(cid:11)ee urn."A picture of the agent in its starting situation, when it is given the instruction,is shown in Plate 6.Steps are (cid:12)rst processed by a parser that uses a combinatory categorialgrammar (CCG) [Ste90] to produce an action representation based on Jack-endo(cid:11) ’s Conceptual Structures [Jac90]. We are using CCG because of itsfacility with conjoined constituents, which are common in instructions { forexample\Clear and rope o(cid:11) an area around the aircraft and post warningsigns." [Air Force Maintenance manual T.O. 1F-16C-2-94JG-50-2]234CHAPTER 6. TASK-LEVEL SPECIFICATIONSFigure 6.2: AnimNL System Architecture.6.4. A MODEL FOR INSTRUCTION UNDERSTANDING235We are using Jackendo(cid:11) ’s Conceptual Structures for two reasons: (cid:12)rst, theprimitives of his decompositional theory capture important generalizationsabout action descriptions and their relationships to one another, and second,they reveal where information may be missing from an utterance and have tobe provided by inference. For the instruction step \Go into the kitchen to getme the co(cid:11)ee urn", the parser produces the following structure:GO([Sp]i;[TO([IN([])])])agentkitchen(cid:20)FOR()(cid:12)(cid:21)(cid:11)[CAUSE([GO([Spi;]j)])]; k(cid:12)coffee-urnFROM([AT()])j(cid:20)(cid:21)TO()lkThis representation makes explicit the fact that getting the co(cid:11)ee urn involvesits moving from its current location to a new one (which should be the locationof the instructor). The FOR-function (derived from the to -phrase) encodesthe purpose relation holding between the go -actionand the get -action.(cid:11)(cid:12)Indices indicate di(cid:11)erent instances of a single conceptual type [ZV92].From these indexed conceptual structures, an initial plan graph is con-structed to represent the agent’s intentions, beliefs and expectations aboutthe task it is to perform. To do this, the system consults the agent’s knowl-edge of actions and plans (the Action KB and Plan Library in Figure 6.2), todevelop hypotheses about the instructor-intended relationships between thespeci(cid:12)ed actions (e.g., temporal relations, enablement relations, generationrelations, etc.). The initial plan graph for our running example is shown inFigure 6.3.ampleThis initial plan graph is further elaborated through processes of referenceresolution, plan inference, reference grounding, plan expansion and perfor-mance (through simulation). To show the interaction between these processesand how they are used to elaborate the plan graph, we will contrast our ex-\Go into the kitchen to get me the co(cid:11)ee urn."with a somewhat di(cid:11)erent but related example\Go into the kitchen and wash out the co(cid:11)ee urn."In the (cid:12)rst case, recall from the conceptual structure produced by theparser, that \get me" is interpreted as an instance of a \cause something togo somewhere" action. One recipe that the system has in its Plan Library foraccomplishing this is shown in Figure 6.4. With respect to this recipe, \go"can be seen as a substep of \get" { which is one way it can serve the purposeof \get".(This action representation and the plan graph are described ingreater detail in [EW92].)Getting an ob ject from one place to another requires (cid:12)rst going to itslocation. This leads to the assumption, noted in Figure 6.3, that the co(cid:11)ee236CHAPTER 6. TASK-LEVEL SPECIFICATIONSA1: BE(urn, IN([other-room]))substep"GET(urn)"substepenabG 2substepenab1"GO.sp(TO(urn))""GO.ctrl(urn)"3"GO.sp.with(urn)"gen
(A1)"GO.sp(INTO(kitchen))"Figure 6.3: Initial Plan Graph: \Go into the kitchen and get me the co(cid:11)eeurn."urn is in the kitchen. (The role of plan inference in instruction understandingis discussed in more detail in [Di 92, DW92].) Reference resolution cannotcontribute any further constraints to the description \the co(cid:11)ee urn", since(1) there is no urn in the discourse context (nor anything that has a uniqueco(cid:11)ee urn associated with it), and (2) the assumption that the urn is in thekitchen is incompatible with its being unique in the current spatio-temporalcontext (which is the room next to the kitchen). Reference grounding doesnot attempt to associate this description with an ob ject in the current spatio-temporal context, for the same reason. In fact, the agent will not attempt toground this referring expression until it has entered the kitchen. (Whetherthe agent then succeeds immediately in grounding the expression will dependon whether the urn is perceivable { i.e., out in full view. We will discuss thisshortly. In any case, the agent expects to be able to get access to the urn whenit gets to the kitchen. This is what will drive it to seek the urn, if it is not inview when it gets to the kitchen.)In the contrasting example \Go into the kitchen and wash out the co(cid:11)eeurn", the system again hypothesizes that the purpose relation between go andwash-out is a substep relation { but in this case, it is because washing out anob ject requires being at a washing site (e.g., a sink or tub). That kitchensusually have sinks gives further weight to this hypothesis.Reference resolution may now contribute something to the agent’s un-derstanding of the de(cid:12)nite expression \the co(cid:11)ee urn". While the discoursecontext does not provide evidence of a unique co(cid:11)ee urn, either directly or byassociation, there is also no evidence against the hypothesis that the urn isin the current spatio-temporal context. An initial hypothesis added by refer-ence resolution that the urn is in the current space, if con(cid:12)rmed by referenceGG6.4. A MODEL FOR INSTRUCTION UNDERSTANDING237Header[CAUSE([[GO()])]]i;Spj; kagentFROM([AT()])j(cid:20)(cid:21)TO()lkBody- [GO([, [TO([AT()])])]Sp(cid:13)1ij- [CAUSE(, [GO(, [TO([AT()])])])]Ctrl(cid:13)2ijiGO()Spi; k-(cid:20)(cid:21)[WITH()]j(cid:13)3--Annotations-enablesenables(cid:13)2(cid:13)3(cid:13)1- [NOT BE(,)]SpjlQuali(cid:12)ersE(cid:11)ects- [BE(,)]SpjlFigure 6.4: A Move Something Somewhere Action.grounding, would lead plan expansion (through sub-goal generation) to getthe agent over to its location. Failure of that hypothesis would lead to thealternative hypothesis that the co(cid:11)ee urn is in the kitchen. This is the samehypothesis as in the original example { it has just arisen in a di(cid:11)erent way.The next thing to discuss is how the plan graph is expanded, and why it isexpanded incrementally, as actions are performed in accordance with earlierelements of the plan graph. How it is expanded is through subgoal generationdown to what we have called annotated task actions. This process makes useof a new kind of planner that (1) eschews pre-conditions in favor of decisionsbased on the agent’s positive and negative intentions, and (2) takes upcomingintentions into account when deciding how to expand current goals, so as toput the agent in the best position with respect to satisfying those intentions.This planner, called ItPlanS, is described in more detail in [Gei92]. It is also238CHAPTER 6. TASK-LEVEL SPECIFICATIONSFigure 6.5: The Urn is Not Visible, so Cabinets will be Opened.the source of the annotations of purpose in annotated task actions.The main reason why the plan graph is expanded incrementally is that theagent does not have su(cid:14)cient knowledge, before beginning to act, of what itwill need to do later. In particular, AnimNL assumes that an agent cannothave up-to-date know ledge of any part of its environment that is outside itsdirect perception. (An AnimNL agent may know what non-visible parts of itsenvironment were like, when it saw them earlier, and have expectations aboutwhat they wil l be like, when it sees them next, but its know ledge is limitedto general truths about the world and to its direct perceptions.) As for theextent of the agent’s perception, it is assumed that an agent cannot see intoany space that has no portal open into the space the agent occupies. ThusAnimNL agents have to open doors, closets, boxes, etc., if they want to knowwhat is inside, or go into other rooms to (cid:12)nd out what is there.What this means in our example is that only the plan graph node corre-sponding to \go into the kitchen" can be expanded { in this case, to \go overto the door", \open door", and \enter kitchen" { before the agent begins toact. The node corresponding to \go to the location of the co(cid:11)ee urn" cannotbe expanded until the door has been opened and the agent can see whetheror not the urn is visible. If it is visible, the agent can go to its location (Plate6). If it is not visible, this same node must be expanded with actions cor-responding to (cid:12)nding the urn { going through the kitchen cabinets one at atime looking for the urn, until it is found or all cabinets have been searched(Figure 6.5).When an annotated task action becomes su(cid:14)ciently speci(cid:12)ed for the agenttion 6.3).time.)following:it one way;6.4. A MODEL FOR INSTRUCTION UNDERSTANDING239to be ready to commit to it and temporal dependencies permit such commit-ment, it is gated, triggering other, low-level planning processes (see Figure 6.2below the \action gate"). An annotated task action is su(cid:14)ciently speci(cid:12)ed if(cid:15) the action is \executable" (i.e., a task action, as described in Sec-(cid:15) all actions temporally prior to it have been committed to. (Note thatprevious actions need not be completed before a new action is committedto: an agent can be (and usually is) doing more than one thing at a(cid:15) its purpose has been determined.It is worthwhile saying a bit more here about these purpose annotations,since we have come to believe they play a large part in low-level decisionsabout how to act. The kind of observations that motivates them are the(cid:15) when told to pick up a book and hand it to someone, an agent will grasp(cid:15) when told to pick up the same book and turn it over, an agent willcommonly grasp it in quite a di(cid:11)erent way;(cid:15) when told to pick up the book and open to page 70, the agent will graspit yet a third way.(cid:15) when just told to pick up the book, and nothing further, agents com-monly grasp it, lift it up and wait expectantly for the next command.These variations in grasp extend to such low-level features as grasp site andwrist position.What we have tentatively concluded from such observations is that whenagents don’t know the purpose of some action they are told to perform, theyput themselves into a position that easily supports subsequent action. Ofcourse, always going into a position in which an agent is poised for subse-quent action is very ine(cid:14)cient, especially when the agent knows what thatsubsequent action will be.In that case, he or she acts in such a way tosmoothly and e(cid:14)ciently transition from one to the other. In AnimNL, pur-pose annotations (including \PFA" or poised for action) are there to allow thesimulator, upon action commitment, to come up with the most e(cid:11)ective waysof moving the agent’s body for the given purpose. It is also why the system isdesigned to delay commitment until it knows the purpose of any task actionor knows that the only thing it can know is PFA.When an action is committed to, there is still further work to be donein order to determine the agent’s behavior. In particular, one result of theexperiment described in the previous section (Section 6.3) was our recognitionof the need for tailoring an agent’s behavior in carrying out an action to the240CHAPTER 6. TASK-LEVEL SPECIFICATIONStype of ob ject given as an argument to that action. This follows from the factthat the same Natural Language verb is commonly used with di(cid:11)erent ob jectsto denote very di(cid:11)erent behavior on an agent’s part, and for a task animationto be correct, these di(cid:11)erences must be depicted.Consider, for example, the following de(cid:12)nition (from [JCMM73]) of theword \remove" and sentences illustrating its use:Remove: to perform operations necessary to take an equipment unitout of the next larger assembly or system;to take o(cid:11) or eliminate; to take or move away.1a. Remove bleed air shuto(cid:11) valves.1b. Remove bolts from nuts.2. Remove paint.3. Remove covers.For each di(cid:11)erent ob ject, the behavior needed to e(cid:11)ect a \remove" is quitedi(cid:11)erent. The question is whether to de(cid:12)ne a single remove-action, to usein animating both Remove the paint and Remove the bolt? The alternative {de(cid:12)ning a multitude of animation procedures (e.g. remove-paint, remove-bolt,remove-nut, remove-nail, remove-boxtop, etc.) { appears expensive in termsof time and e(cid:11)ort, and prone to error.The solution we are adopting is to build a hybrid system.Instead ofspecifying complete de(cid:12)nitions for each verb, we can identify the core or ker-nel action for a verb like remove in a fashion similar to that described inSection 6.2. We will use this core meaning, central to many di(cid:11)erent instan-tiations of the verb, in building the task-action. The missing information canbe supplied by the verb’s ob ject: The knowledge base is ob ject-oriented andso can store relevant information about individual ob jects. For example, oneslot of information might be the DOFs an ob ject has { a bolt \knows" (i.e., itsgeometric constraints specify) around which axis it turns. Joint and rotationinformation is already available in Jack.The hybrid system would process an instruction by combining the in-formation in the two representations { the underspeci(cid:12)ed de(cid:12)nitions of thetask-actions, in conjunction with the ob ject-oriented knowledge base. Byidentifying which information is lacking in the task-actions, the system cantry to supply that information from the knowledge base.The advantages of a hybrid system is economy of both action de(cid:12)nitionsand of the ob ject feature information to be stored. We no longer need to worryabout developing separate de(cid:12)nitions for each animation movement based ondistinct verb/ob ject pairs. Instead we take advantage of the compositionalnature of the task-actions, and the ob ject-oriented, hierarchical knowledge-base. Using these utilities, we can de(cid:12)ne a single animation de(cid:12)nition forremove which will allow us to animate both Remove the bolt and Remove nutsfrom bolts while still distinguishing the instruction Remove covers.Our work on using complex Natural Language instructions to motivate thebehavior of animated agents is still in its infancy. There is much more to bedone before it is a useful tool in the hands of task designers and human factors6.4. A MODEL FOR INSTRUCTION UNDERSTANDING241engineers. On the other hand, we have begun to demonstrate its potential(cid:13)exibility in accommodating the task behavior of an agent to the environmentin which the task is being carried out and the agent’s own capabilities.242CHAPTER 6. TASK-LEVEL SPECIFICATIONSChapter 7Epilogue1To de(cid:12)ne a future for the work described in this book, it is essential to keepin mind the broad goals which motivated the e(cid:11)orts in the (cid:12)rst place. Usefuland usable software is desired, to be sure, but the vision of manipulating andespecially instructing a realistically behaved animated agent is the greaterambition. Some of our visions for the near future are presented, not just forthe sake of prognostication, but for its exciting prospects and possibilities.Any discussion of the future of software must take into account the ex-traordinary pace of developments in the hardware arena. Even conservativepredictions of hardware capabilities such as speed and capacity over the (cid:12)veyear term lead one perilously close to science (cid:12)ction. Accordingly, predictionsof \better, faster, cheaper, more reliable, more fault tolerant, more highly par-allel computers" are easy to make but do little to inform us of the applicationsthese fantastic machines will facilitate. Rather, as general purpose comput-ers improve in all these ways, specialized hardware solutions will decreasein importance and robust, usable software and symbiotic human-computerinterfaces will remain the crucial link between a task and a solution.Transforming research into practice is a lengthy process, consisting of a(cid:13)ow of concepts from ideas through algorithms to implementations, from test-ing and analysis through iterated design, and (cid:12)nally transfer of demonstrat-ably workable concepts to external users and actual applications. This entireprocess may span years, from the initial description of the concept to a (cid:12)eldedsystem. The publication of initial results often breeds over-optimism and hasbeen known to lead researchers to allow false expectations to arise in the mindsof potential users, with unfortunate results. (Automatic machine translationof text, speech understanding, and early promises of Arti(cid:12)cial Intelligenceproblem solving are good examples of premature speculations.) At the otherend of the spectrum, however, are concepts which take a long time to worktheir way into mainstream technological consciousness. (3D computer graph-ics is a good example where concepts and even working systems pre-dated1With the help of Mark Steedman.243244CHAPTER 7. EPILOGUEwidespread commercial availability by more than a decade.) So we will at-tempt to strike a balance in making speculations: while looking toward a longterm research plan we will generally consider technology transfer to occurwhen serious but sympathetic users can experiment and accomplish real workwith it. Our experience with software in the past is both our model and ourpromise for expecting new concepts to eventually reach potential users forevaluation and feedback.7.1 A Roadmap Toward the FutureWe seek to study the conceptual structure and limits of \virtual agents" in\simulated tasks" (VAST): the software and interface systems necessary topermit a user to describe, control, animate, analyze, interact with, and co-operate with multiple virtual computer-synthesized human models. We willremain cognizant of anticipated developments in underlying computer capa-bilities, but our principal intention will be to probe the intelligent softwareand user-interface issues. VAST focuses on the simulated human (cid:12)gure notjust as a graphical entity but as an active, behaviorally complex agent whocan follow instructions and autonomously negotiate its own way in the world.Recall that our introduction emphasized certain simulation goals:(cid:15) Create an interactive computer graphics human model;(cid:15) Endow it with reasonable biomechanical properties;(cid:15) Provide it with \human-like" behaviors;(cid:15) Use this simulated human as an agent to e(cid:11)ect changes in its world;(cid:15) Describe and guide its tasks through natural language instructions;VAST augments this list to further improve the symbiosis between user andvirtual agents:(cid:15) Control the agent through natural manual interfaces;(cid:15) Automatically generate explications or commentary on its behavior assub-titles (text), voice-over or its own speech;(cid:15) Coordinate the activity of multiple agents engaged in a task.We have probed the state-of-the-art in the (cid:12)rst set of goals, but many prob-lems and prospects remain for study. Indeed, there are other e(cid:11)orts that areadvancing human modeling and animation. But our emphasis on interactivityand usability, especially by non-animators, outweighs mere visual beauty.7.1. A ROADMAP TOWARD THE FUTURE2457.1.1 Interactive Human ModelsWhile we are currently partway toward this goal with Jack, there are enhance-ments that are necessary before a virtual human looks and behaves realisti-cally. Increases in hardware capability will certainly aid in the presentationof highly detailed models with smooth real-time response. The realistic andbeautiful human models created by the Thalmanns [MTT90, MTT91a] areillustrative of the surface veracity possible under non-real-time conditions.7.1.2 Reasonable Biomechanical PropertiesJoint limits prevent unnatural adjacent body segment postures, but do noth-ing to prevent non-adjacent collisions. Collision-avoidance should be an im-plicit part of interactive manipulation for body integrity and natural appear-ance. In addition, clothing or equipment worn by the virtual human shoulddemonstrate similar collision-free behavior.Realistic strength models for the whole body and especially the torsoshould be incorporated into the model not only as a source of data but asan active resource for body motions. Preliminary work on strength-guidedmotion has shown feasibility, but more study of joint torque distributionstrategies under comfort and loading constraints is needed. Ongoing \per-formance" models of the virtual human should be maintained throughout aninteractive session or animation so that realistic assessments of workload andfatigue may be monitored.7.1.3 Human-like BehaviorsWe must continue to build a \primitive" behavior library so that the \innate"motion vocabulary of the virtual (cid:12)gure is as broad as possible. Ideally thebehaviors can be \taught" to the (cid:12)gure rather than procedurally coded. Eachbehavior should enumerate the consequences of its execution so that higherlevel planning activities may take its potential e(cid:11)ects into account.Posture planning should take into account the spatial organization of theworld and the virtual agent’s desire to maximize e(cid:11)ective behavior while min-imizing useless movements (work). This can be done, in part, by havinge(cid:11)ective collision-avoidance schemes for articulated (cid:12)gures and, in part, byusing symbolical spatial information to prune the high-dimensional numericalsearch space and move the (cid:12)gure into predictably useful postures. We alreadyrealize that classical AI planning paradigms are too weak for posture planning,and more reactive, incremental planners with \mental motion simulation" areneeded.Moving a virtual (cid:12)gure around an environment requires more than simplelocomotion behavior: the behavioral repertoire and planner should under-stand crawling, climbing, jumping, sliding, etc. With this large repertoire ofpossible behaviors, a planner will be busy coordinating them all and sortingout priorities, even for simple activities.246CHAPTER 7. EPILOGUE7.1.4 Simulated Humans as Virtual AgentsThe notion of skill level should be quanti(cid:12)ed with notions of context-sensitiveexecution time and [optimal] motion smoothness. Synthesized animationsshould be customized to the user’s own body size and capabilities (or limita-tions) for training situations.The environment in which the virtual humans work must be importedfrom any number of external CAD systems, preferably through standardizedinterfaces. During importation, perceptually and behaviorally signi(cid:12)cant fea-tures such as articulations, handles, removable parts, and open spaces (holes)should be recognized from the geometric model. We can expect some CADsystems to o(cid:11)er some of this data, but in general we should expect to buildenhanced semantics into the models interactively or semi-automatically our-selves. For example, handles are needed to determine likely grasp points,and articulations, parts, and holes are needed for automatic generation ofdisassembly behaviors given only the ob ject descriptions.7.1.5 Task Guidance through InstructionsFor a designer to use Natural Language instructions to describe and guidea virtual agent through a task, the overall system must know how to un-derstand instructions and to use them appropriately in a given environment.Instructions must be understood in terms of intention { what is meant to beachieved in the world { and in terms of positive and negative constraints onthe behavior used to achieve it. Instructions must be used to interpret fea-tures of the environment and to coordinate the agent’s task-level knowledgeand skills in acting to achieve its given goals. Advances in AI planning andexecution are coming at a rapid rate, independent of our own work, and wewill be incorporating those advances into VAST, to make the bridge to actualbehavior.7.1.6 Natural Manual Interfaces and Virtual RealityThe virtual (cid:12)gure should exist in a virtual 3D world that is accessible to auser with a minimum of training and little, if any, computer expertise. Whilemouse and keyboard input to Jack addresses some of these goals, it is stilltoo \low level." By taking advantage of novel 3D, 6D, multiple 6D, and handposture sensor input devices, the user’s movements can be translated directlyinto virtual (cid:12)gure behaviors. The trick is not to make the mapping one-to-one, so the user exhausts herself (cid:13)ailing arms and twisting her body as thecurrent Virtual Reality paradigms would have one do. Rather the mappingshould have considerable intelligent \multipliers" so that the suggestion ofaction is enough to precipitate complete behaviors in the virtual human. Wecan already control the center of mass of the (cid:12)gure to e(cid:11)ect a signi(cid:12)cantmultiplier of input e(cid:11)ort, this needs to be extended to arm gestures, viewfocus, and locomotion generation. We envision a strong corroborating role7.1. A ROADMAP TOWARD THE FUTURE247from our animation from instructions work, such as speech-based commands.Minimal device encumbrances on the user are deemed essential.7.1.7 Generating Text, Voice-over, and Spoken Explica-tion for AnimationAnimation for purposes of instruction in the performance of a task frequentlyrequires spoken or written text, as well as graphic presentation, if it is tobe understood. Negative instructions such as \warnings" or \cautions" pro-vide an obvious example. While written captions can be adequate for somepurposes, presenting few problems for animation, spoken language is moree(cid:14)cient and more engaging of the viewers’ attention. For certain purposeseven, a human face speaking in accompaniment to action is the most e(cid:14)cientand attention-holding device of all. Among the problems involved in includinglinguistic information in an on-going animation are: appropriately allocatinginformation to the di(cid:11)erent modalities; integration of the two modalities overtime; limitations of existing speech synthesizers with respect to intonation;integration of a facial animation with speech. All of these are tasks which theanimator can in principle take over, but all of them, especially the last, arelaborious. It would be highly desirable to automate all of them, especially ifthey are to be used in highly interactive animations, in which model-basedsynthesis-by-rule is required, rather than image based techniques.One of the extensions to Jack is an animated facial model with a program-ming language for integrating its movements with a segmental representationof speech [PBS91, Pel91]. This e(cid:11)ort is now focused on the following exten-sions:speech.(cid:15) Provide an improved programming language for facial animation of(cid:15) Provide a discourse semantics for spoken intonation in terms of appro-priate knowledge representations.(cid:15) Perform automatic generation from such semantic representations ofphonological representations of spoken explications, including appropri-ate intonational markers of contrast and background, for input to aspeech synthesizer, with or without the facial animation program.7.1.8 Coordinating Multiple AgentsOur principle goal has been getting a single agent to behave plausibly in re-sponse to multi-clause instruction steps. One goal for the longer term involvesproducing sensible behavior on the part of a single virtual agent and plausiblebehavior from a group of virtual agents engaged in a multi-agent task.Coordinated multi-person simulations will be the next [large] step afteran individual agent’s actions can be e(cid:11)ectively determined. Such simula-tions require physical, task and cognitive coordination among the agents. For248CHAPTER 7. EPILOGUEphysical coordination, our particular e(cid:11)orts will focus on determining timingcoordination, strength and workload distribution, and mutual achievement ofspatial goals. In these multi-agent tasks, the interactive user may or may notbe one of the participating agents.Task coordination requires augmenting our task knowledge base with in-formation on multi-agent tasks. This is essential both for understanding thetext of multi-agent task instructions and for interpolating between explicitinstructions, since instructions cannot (by virtue of not knowing the precisecircumstances under which they will be carried out) specify everything.For cognitive coordination among the agents, communication may be nec-essary to determine or alter the leadership role, initiate activity, keep it mov-ing along, interrupt or abort it, or rest on imminent fatigue. Research oncommunication for coordinating multi-agent tasks is being carried on at otherinstitutions [GS89, CL91, Loc91]. In the longer term, we look to importing theresults of this research and incorporating it into the VAST framework. Untilthen, we will focus on single virtual agents or centrally-controlled multipleagents engaged in tasks in which communication is not required for coordina-tion.7.2 ConclusionThere are a multitude of other directions for virtual agent work. Some ofthese are automatic view control, a perceptual \sense," spatial reasoning forimproved posture planning, recognizing and accommodating task failures, skillacquisition, (cid:13)exible ob ject interactions, animation presentation techniques,behaving with common sense, enriched instruction understanding, and speech-based agent animation. But all that’s for a sequel.Bibliography[AAW74] M. A. Ayoub, M. M. Ayoub, and A. Walvekar. A biomechanical modelfor the upper extremity using optimization techniques. Human Factors,[Abb53]Edwin A. Abbott. Flatland; a romance of many dimensions. Dover,16(6):585{594, 1974.New York, NY, 1953.[ABS90]T. Alameldin, N. Badler, and T. Sobh. An adaptive and e(cid:14)cient systemfor computing the 3-D reachable workspace.In Proceedings of IEEEInternational Conference on Systems Engineering, pages 503{506, 1990.[AC87]Phillip Agre and David Chapman. Pengi: An implementation of atheory of activity. Proceedings of the AAAI-87 Conference, pages 268{272, June 1987.[AD90]T.L. Anderson and M. Donath. Animal behavior as a paradigm fordeveloping robot autonomy.In Pattie Maes, editor, Designing Au-tonomous Agents, pages 145{168. MIT Press, 1990.[AG85]W. W. Armstrong and Mark Green. The dynamics of articulated rigidbodies for purposes of animation. The Visual Computer, 1(4):231{240,[AGL87] William Armstrong, Mark Green, and R. Lake. Near-real-time controlof human (cid:12)gure models. IEEE Computer Graphics and Applications,1985.7(6):52{61, June 1987.[AGR81] M. M. Ayoub, C. F. Gidcumb, M. J. Reeder, M. Y. Beshir, H. A. Hafez,F. Aghazadeh, and N. J. Bethea. Development of an atlas of strengthsand establishment of an appropriate model structure. Technical Report(Final Report), Institute for Ergonomics Research, Texas Tech Univ.,Lubbock, TX, Nov. 1981.[AGR82] M. M. Ayoub, C. F. Gidcumb, M. J. Reeder, H. A. Hafez, M. Y. Beshir,F. Aghazadeh, and N. J. Bethea. Development of a female atlas ofstrengths. Technical Report (Final Report), Institute for ErgonomicsResearch, Texas Tech Univ., Lubbock, TX, Feb. 1982.[AHN62]E. Asmussen and K. Heeboll-Nielsen. Isometric muscle strength in re-lation to age in men and women. Ergonomics, 5(1):167{169, 1962.[Ala91]Tarek Alameldin. Three Dimensional Workspace Visualization for Re-dundant Articulated Chains. PhD thesis, Computer and InformationScience, Univ. of Pennsylvania, Philadelphia, PA, 1991.[Alb81]James S. Albus. Brains, Behavior, and Robotics. BYTE Books,McGraw-Hill, 1981.[Ali90]Alias Research, Inc. ALIAS V3.0 Reference Manual, 1990.[AM71]B. Anson and C. McVay. Surgical Anatomy. Saunders, Philadelphia,++249250BIBLIOGRAPHY[And60]E. Anderson. A semigraphical method for the analysis of complex prob-lems. Technometrics, 2:381{391, 1960.[And72]D. F. Andrews. Plots of high-dimensional data. Biometrics, 28(125),PA, 1971.1972.[Ayo91]M. Ayoub. From biomechanical modeling to biomechanical simulation.In Edward Boyle, John Ianni, Jill Easterly, Susan Harper, and Med-hat Korna, editors, Human-Centered Technology for Maintainability:Workshop Proceedings. Wright-Patterson Air Force Base, ArmstrongLaboratory, June 1991.[Bad75]Norman I. Badler. Temporal scene analysis: Conceptual descriptionsof object movements. PhD thesis, Computer Science, Univ. of Toronto,Toronto, Canada, 1975. (Univ. of Pennsylvania, Computer and Infor-mation Science, Tech. Report MS-CIS-76-4).[Bad76]Norman I. Badler. Conceptual descriptions of physical activities. Amer-ican Journal of Computational Linguistics, Micro(cid:12)che 35:70{83, 1976.[Bad89]Norman I. Badler. A representation for natural human movement. InJ. Gray, editor, Dance Technology I, pages 23{44. AAHPERD Publica-tions, Reston, VA, 1989.[Bar89]David Bara(cid:11). Analytical methods for dynamic simulation of non-penetrating rigid bodies. Computer Graphics, 23(3):223{232, 1989.[BB78]Norman I. Badler and Ruzena Ba jcsy. Three-dimensional representa-tions for computer graphics and computer vision. Computer Graphics,12(3):153{160, Aug. 1978.[BB88]Ronen Barzel and Alan H. Barr. A modeling system based on dynamicconstraints. Computer Graphics, 22(4):179{188, 1988.[BBA88]P. G. Bullough and O. Boachie-Adjei. Atlas of Spinal Diseases. Lippin-cott, Philadelphia, PA, 1988.[BBB87]Richard H. Bartels, John C. Beatty, and Brian A. Barsky. An Introduc-tion to Splines for Use in Computer Graphics and Geometric Modeling.Morgan Kaufmann, Los Altos, CA, 1987.[BBH90] C. Blanchard, S. Burgess, Y. Harvill, J. Lanier, A. Lasko, M. Oberman,and M. Teitel. Reality built for two: A virtual reality tool. ComputerGraphics, 24(2):35{36, 1990.[BC68]R. Beckett and K. Chang. An evaluation of the kinematics of gait byminimum energy. Journal of Biomechanics, 1:147{159, 1968.[BC89]Armin Bruderlin and Tom W. Calvert. Goal-directed, dynamic anima-tion of human walking. Computer Graphics, 23(3):233{242, 1989.[BCS90]R. D. Beer, H. J. Chiel, and L. S. Sterling. A biological perspectiveon autonomous agent design.In Pattie Maes, editor, Designing Au-tonomous Agents, pages 169{186. MIT Press, 1990.[Bec92]Welton M. Becket. Simulating adaptive autonomous behavior with re-current neural networks. Technical report, Computer and InformationScience, Univ. of Pennsylvania, Philadelphia, PA, 1992. To appear.[BEK81] P. Bapu, S. Evans, P. Kitka, M. Korna, and J. McDaniel. User’sguide for COMBIMAN programs. Technical Report AFAMRL-TR-80-91, Univ. of Dayton Research Institute, Jan 1981. U.S.A.F. Report.[Ber83]J. Bertin. Semiology of Graphics, translated by W. J. Berg. The Univ.of Wisconsin Press, 1983.[BG86]Norman I. Badler and Je(cid:11)rey S. Gangel. Natural language input for hu-++BIBLIOGRAPHY251man task description. In Proc. ROBEXS ’86: The Second InternationalWorkshop on Robotics and Expert Systems, pages 137{148. InstrumentSociety of America, June 1986.+[BHJ83] Michael Brady, John M. Hollerbach, Timothy L. Johnson, TomasLozano-P(cid:19)erez, and Matthew T. Mason, editors. Robot Motion: Plan-ning and Control. MIT Press, Cambridge, MA, 1983.[Bie86]Eric Allan Bier. Snap-dragging. Computer Graphics, 20(3):233{240,[Bie87]Eric Allan Bier. Skitters and jacks: Interactive positioning tools. In Pro-ceedings of 1986 ACM Workshop on Interactive 3D Graphics, ChapelHill, NC, Oct. 1987.[Bie90]Eric Allan Bier. Snap-dragging in three dimensions. Computer Graph-ics, 24(2):193{204, March 1990.+[BKK85] Norman I. Badler, Jonathan Korein, James U. Korein, Gerald Radack,and Lynne Brotman. Positioning and animating human (cid:12)gures in atask-oriented environment. The Visual Computer, 1(4):212{220, 1985.[BKT86]K. Bo(cid:11), L Kaufmann, and J Thomas, editors. The Handbook of Percep-tion and Human Performance. John Wiley and Sons, New York, NY,1986.1986.[BL88]Kenneth R. Bo(cid:11) and Janet E. Lincoln, editors. Engineering Data Com-pendium. Harry G. Armstrong Aerospace Medical Research Laboratory,Wright-Patterson Air Force Base, OH, 1988.[BL89]J. Barraquand and J. Latombe. Robot motion planning: A distributedrepresentation approach. Technical Report STAN-CS-89-1257, Com-puter Science, Stanford Univ., Stanford, CA, May 1989.[Bli82]James F. Blinn. A generalization of algebraic surface drawing. ACMTransactions on Graphics, 1(3):235{256, July 1982.[BLL89a]J. Barraquand, B. Langlois, and J. Latombe. Numerical potential (cid:12)eldtechniques for robot path planning. Technical Report STAN-CS-89-1285, Computer Science, Stanford Univ., Stanford, CA, 1989.[BLL89b]J. Barraquand, B. Langlois, and J. Latombe. Robot motion planningwith many degrees of freedom and dynamic constraints. In Fifth Intl.Sym. on Robotics Research (ISRR), Tokyo, pages 1{10, 1989.[BLP78]Edward G. Britton, James S. Lipscomb, and Michael E. Pique. Makingnested rotations convenient for the user. Computer Graphics, 12(3):222{227, August 1978.[BLP83]Rodney A. Brooks and Tomas Lozano-P(cid:19)erez. A subdivision algorithmin con(cid:12)guration space for (cid:12)ndpath with rotation. In Proc. 8th Int. JointConf. Arti(cid:12)cial Intel ligence, pages 799{806, 1983.[BMB86]Norman I. Badler, Kamran H. Manoochehri, and David Bara(cid:11). Multi-dimensional input techniques and articulated (cid:12)gure positioning by mul-tiple constraints. In Proc. Workshop on Interactive 3D Graphics, NewYork, NY, Oct. 1986. ACM.[BMTT90] R. Boulic, Nadia Magnenat-Thalmann, and Daniel Thalmann. A globalhuman walking model with real-time kinematic personi(cid:12)cation. TheVisual Computer, 6:344{358, 1990.[BMW87] Norman I. Badler, Kamran Manoochehri, and G. Walters. Articulated(cid:12)gure positioning by multiple constraints.IEEE Computer Graphicsand Applications, 7(6):28{38, 1987.[BN88]L. S. Brotman and A. N. Netravali. Motion interpolation by optimal252BIBLIOGRAPHYcontrol. Computer Graphics, 22(4):309{315, 1988.[Bob88]J. E. Bobrow. Optimal robot path planning using the minimum-timecriteria. IEEE Journal of Robotics and Automation, 4(4):443{450, Au-[Bod77]Margaret Boden. Arti(cid:12)cial Intel ligence and Natural Man. Basic Books,[BOK80]Norman I. Badler, Joseph O’Rourke, and Bruce Kaufman. Special prob-lems in human movement simulation. Computer Graphics, 14(3):189{gust 1988.New York, NY, 1977.197, July 1980.[BOT79]Norman I. Badler, Joseph O’Rourke, and Hasida Toltzis. A spheri-cal representation of a human body for visualizing movement. IEEEProceedings, 67(10):1397{1403, Oct. 1979.[BP88]Alain Berthoz and Thierry Pozzo.Intermittent head stabilizationduring postural and locomotory tasks in humans.In B. Amblard,A. Berthoz, and F. Clarac, editors, Posture and Gait: Development,Adaptation, and Modulation. Excerpta Medica, 1988.[Bra84]Valentino Braitenberg. Vehicles: Experiments in Synthetic Psychology.The MIT Press, 1984.[Bre89]David E. Breen. Choreographing goal-oriented motion using cost func-tions. In N. Magnenat-Thalmann and D. Thalmann, editors, State-of-the-Art in Computer Animation, pages 141{151. Springer-Verlag, NewYork, NY, 1989.[Bro83a]Rodney A. Brooks. Planning collision-free motions for pick-and-placeoperations. Int. Journal of Robotics Research, 2(4):19{44, Winter 1983.[Bro83b]Rodney A. Brooks. Solving the (cid:12)nd-path problem by good representa-tion of free space. IEEE Transactions on Systems, Man and Cybernet-ics, SMC-13(3):190{197, Mar 1983.[Bro86]Rodney A. Brooks. A robust layered control system for a mobile robot.IEEE Journal of Robotics and Automation, pages 14{23, April 1986.[Bro90]Rodney A. Brooks. Elephants don’t play chess. In Pattie Maes, editor,Designing Autonomous Agents, pages 3{18. MIT Press, 1990.[Bru88]Armin Bruderlin. Goal-directed, dynamic animation of bipedal locomo-tion. Master’s thesis, Simon Fraser Univ., Vancouver, Canada, 1988.[BS76]Maxine Brown and Stephen W. Smoliar. A graphics editor for Laban-otation. Computer Graphics, 10(2):60{65, 1976.[BS79]Norman I. Badler and Stephen W. Smoliar. Digital representations ofhuman movement. ACM Computing Surveys, 11(1):19{38, March 1979.[BS91]Jules Bloomenthal and Ken Shoemake. Convolution surfaces. ComputerGraphics, 25(4):251{256, 1991.[BSOW78] Norman I. Badler, Stephen W. Smoliar, Joseph O’Rourke, and LynneWebber. The simulation of human movement. Technical Report MS-CIS-78-36, Computer and Information Science, Univ. of Pennsylvania,Philadelphia, PA, 1978.[BW76]N. Burtnyk and M. Wein. Interactive skeleton techniques for enhancingmotion dynamics in key frame animation. Communications of the ACM,19(10):564{569, Oct. 1976.[BW90]Aijaz A. Baloch and Allen M. Waxman. A neural system for behavioralconditioning of mobile robots. IEEE International Joint Conference onNeural Networks, 2:723{728, 1990.[BwDL80]Irmgard Bartenie(cid:11) and (with Dori Lewis). Body Movement: CopingBIBLIOGRAPHY253with the Environment. Gordon and Breach, New York, NY, 1980.[BWKE91] Norman I. Badler, Bonnie L. Webber, Jugal K. Kalita, and Je(cid:11)reyEsakov. Animation from instructions. In Norman I. Badler, Brian A.Barsky, and David Zeltzer, editors, Making Them Move: Mechanics,Control, and Animation of Articulated Figures, pages 51{93. Morgan-[CA84]D. B. Cha(cid:14)n and G. B. J. Andersson. Occupational Biomechanics.Kaufmann, San Mateo, CA, 1991.John Wiley & Sons, 1984.[Cal91]Tom Calvert. Composition of realistic animation sequences for multi-ple human (cid:12)gures. In Norman I. Badler, Brian A. Barsky, and DavidZeltzer, editors, Making Them Move: Mechanics, Control, and Ani-mation of Articulated Figures, pages 35{50. Morgan-Kaufmann, SanMateo, CA, 1991.[Car72]Sven Carlsoo. How Man Moves. William Heinemann Ltd, 1972.[Cat72]Edwin Catmull. A system for computer generated movies. In Proceed-ings of ACM Annual Conference, pages 422{431, August 1972.[Cat78]E Catmull. The problems of computer-assisted animation. ComputerGraphics, 12(3):348{353, August 1978.[CB92]Wallace Ching and Norman I Badler. Fast motion planning for anthro-pometric (cid:12)gures with many degrees of freedom. In IEEE Intl. Conf. onRobotics and Automation, May 1992.[CBR]K. Corker, A. Bejczy, and B. Rappaport. Force/Torque Display ForSpace Teleoperation Control Experiments and Evaluation. Cambridge,[CCP80]Tom Calvert, J. Chapman, and A. Patla. The integration of sub jectiveand ob jective data in the animation of human movement. ComputerGraphics, 14(3):198{203, July 1980.[CCP82]Tom Calvert, J. Chapman, and A. Patla. Aspects of the kinematicsimulation of human movement. IEEE Computer Graphics and Appli-cations, 2(9):41{50, Nov. 1982.[Ceb87]David Cebula. The semantic data model and large information require-ments. Technical Report MS-CIS-87-72, Computer and InformationScience, Univ. of Pennsylvania, Philadelphia, PA, 1987.[Cen81]NASA Johnson Space Center. Space Shuttle Flight Data File Prepa-ration Standards. Flight Operations Directorate, Operations Division,MA.1981.[Che73]H. Cherno(cid:11). The use of faces to represent points in k-dimensional spacegraphically. J. of the American Statistical Assoc., 68(342), 1973.[CJ71]E. Y. Chao and D. H. Jacobson. Studies of human locomotion viaoptimal programming. Mathematical Biosciences, 6:239{306, 1971.[CL91]P. Cohen and H. Levesque. Teamwork. N^ous, 25, 1991.[CMS88] Michael Chen, S. Joy Mountford, and Abigail Sellen. A study in in-teractive 3-D rotation using 2-D control devices. Computer Graphics,22(4):121{129, August 1988.[Coo68]G.S. Cooper. A semantic analysis of English locative prepositions. Tech-nical Report Report No. 1587, BBN: Clearinghouse for Federal Scienti(cid:12)cand Technical Information, Spring(cid:12)eld, VA, 1968.[Del70]Cecily Dell. A Primer for Movement Description. Dance NotationBureau, Inc., New York, NY, 1970.[DH55]Jacques Denavit and Richard Hartenberg. A kinematic notation for254BIBLIOGRAPHYlower pair mechanisms based on matrices. Journal of Applied Mechan-ics, 23, 1955.[Di 92]Barbara Di Eugenio. Goals and actions in Natural Language instruc-tions. Technical Report MS-CIS-92-07, Computer and Information Sci-ence, Univ. of Pennsylvania, Philadelphia, PA, 1992.[DLRG91] Bruce R. Donald, Jed Lengyel, Mark Reichert, and Donald Greenberg.Real-time robot motion planning using rasterizing computer graphicshardware. Computer Graphics, 25(4):327{336, July 1991.[Don84]Bruce Donald. Motion planning with six degrees of freedom. TechnicalReport 791, MIT AI Lab, 1984.[Don87]B. Donald. A search algorithm for motion planning with six degrees offreedom. Arti(cid:12)cial Intel ligence, 31:295{353, 1987.[Doo82]Marianne Dooley. Anthropometric modeling programs { A survey.IEEE Computer Graphics and Applications, 2(9):17{25, Nov. 1982.[Dru75]C. Drury. Application of Fitts’ Law to foot-pedal design. Human Fac-tors, 17, 1975.[DW92]B. Di Eugenio and B. Webber. Plan recognition in understanding in-structions.In Proc. First Int’l Conference on Arti(cid:12)cial Intel ligencePlanning Systems, Col lege Park MD, pages 52{61, June 1992.[DX89]Bruce Donald and Patrick Xavier. A provably good approximationalgorithm for optimal-time tra jectory planning. In IEEE Intl. Conf. onRobotics and Automation, pages 958{963, 1989.[EB90]Je(cid:11)rey Esakov and Norman I. Badler. An architecture for high-level hu-man task animation control. In P. A. Fishwick and R. S. Modjeski, edi-tors, Know ledge-Based Simulation: Methodology and Application, pages162{199. Springer-Verlag, New York, NY, 1990.[EB91]Je(cid:11)rey Esakov and Norman I. Badler. Animation from instructions {video tape. In Norman I. Badler, Brian A. Barsky, and David Zeltzer,editors, Making Them Move: Mechanics, Control, and Animation ofArticulated Figures. Morgan-Kaufmann, San Mateo, CA, 1991. Video-[EBJ89]Je(cid:11)ery Esakov, Norman I. Badler, and M. Jung. An investigation oflanguage input and performance timing for task animation. In Graph-ics Interface ’89, pages 86{93, San Mateo, CA, June 1989. Morgan-tape.Kaufmann.[EC86]S. M. Evans and D. B. Cha(cid:14)n. Using interactive visual displays topresent ergonomic information in workspace design. In W. Karwowski,editor, Trends in Ergonomics/Human Factors III. Elsevier Science Pub-lishers B.V. (North-Holland), 1986.[EI91]Jill Easterly and John D. Ianni. Crew Chief: Present and future.In Edward Boyle, John Ianni, Jill Easterly, Susan Harper, and Med-hat Korna, editors, Human-Centered Technology for Maintainability:Workshop Proceedings. Wright-Patterson Air Force Base, Armstrong[Emm85]Arielle Emmett. Digital portfolio: Tony de Peltrie. Computer GraphicsLaboratory, June 1991.World, 8(10):72{77, Oct. 1985.[EP87]Ali Erkan Engin and Richard D. Peindl. On the biomechanics of humanshoulder complex { I: Kinematics for determination of the shouldercomplex sinus. Journal of Biomechanics, 20(2):103{117, 1987.[EPE88]S. M. Evans, S. L. Palmiter, and J. Elkerton. The edge system: Er-BIBLIOGRAPHY255gonomic design using graphic evaluation. The Annual Meeting of theHuman Factors Society, Los Angeles, CA, Oct. 1988.[Esa90]Je(cid:11)rey Esakov. KB. Technical Report MS-CIS-90-03, Univ. of Penn-sylvania, Philadelphia, PA, 1990.[ET89]Ali Erkan Engin and S. T. Tumer. Three-dimensional kinematic mod-elling of the human shoulder complex { I: Physical model and deter-mination of joint sinus cones. Journal of Biomechanical Engineering,111:107{112, May 1989.[ETW81] Kenneth B. Evans, Peter Tanner, and Marceli Wein. Tablet basedvaluators that provide one, two or three degrees of freedom. ComputerGraphics, 15(3):91{97, 1981.[Eva85]Susan M. R. Evans. Ergonomics in manual workspace design: Currentpractices and an alternative computer-assisted approach. PhD thesis,Center for Ergonomics, Univ. of Michigan, Ann Arbor, MI, 1985.[Eva88]S. M. Evans. Use of biomechanical static strength models in workspacedesign. In Proceedings for NATO Workshop on Human PerformanceModels in System Design, Orlando, FL, May 1988.[EW92]Barbara Di Eugenio and Michael White. On the interpretation of Nat-ural Language instructions. In Proceedings of 1992 International Con-ference on Computational Linguistics (COLING-92), Nantes, France,[Far88]Gerald Farin. Curves and Surfaces for Computer Aided Geometric De-sign. Academic Press, San Diego, CA, 1988.[Fav84]Bernard Faverjon. Obstacle avoidance using an octree in the con(cid:12)g-uration space of a manipulator. In IEEE Intl. Conf. on Robotics andAutomation, pages 504{512, 1984.[FB85]K. Fishkin and B. Barsky. An analysis and algorithm for (cid:12)lling propa-gation. In Proceedings Graphics Interface, pages 203{212, 1985.[Fet82]William Fetter. A progression of human (cid:12)gures simulated by computergraphics. IEEE Computer Graphics and Applications, 2(9):9{13, Nov.[Fey86]Carl R. Feynman. Modeling the appearance of cloth. Master’s thesis,Massachusetts Institute of Technology, 1986.[Fis86]Paul A. Fishwick. Hierarchical Reasoning: Simulating Complex Pro-cesses over Multiple Levels of Abstraction. PhD thesis, Computer andInformation Science, Univ. of Pennsylvania, Philadelphia, PA, 1986.[Fis88]Paul A. Fishwick. The role of process abstraction in simulation. IEEETransactions on Systems, Man and Cybernetics, 18(1):18{39, Jan/Feb.1992.1982.1988.[Fis90]K. Fishkin. Filling a region in a frame bu(cid:11)er. In A. Glassner, editor,Graphics Gems, pages 278{284. Academic Press, Cambridge, MA, 1990.[Fit54]P. Fitts. The information capacity of the human motor system in con-trolling the amplitude of movement. Journal of Experimental Psychol-ogy, 47:381{391, 1954.[FKU77]H. Fuchs, Z. Kedem, and S. Uselton. Optimal surface reconstructionfrom planar contours. Communications of the ACM, 20(10):693{702,[Fle70]R. Fletcher. A new approach to variable metric algorithms. ComputerOct. 1977.Journal, 13:317{322, 1970.[FLP89]H. Fuchs, M. Levoy, and M. Pizer. Interactive visualization of 3D medi-256BIBLIOGRAPHYcal data. IEEE Transactions on Computers, pages 46{57, August 1989.[FMHR87] S.S. Fisher, M. McGreevy, J. Humphries, and W. Robinett. Virtualenvironment display system. In Proceedings of 1986 ACM Workshopon Interactive 3D Graphics, Chapel Hill, NC, Oct. 1987.[FP64]P. Fitts and J. Peterson.Information capacity of discrete motor re-sponses. Journal of Experimental Psychology, 67(2), 1964.[FS91]James A. Freeman and David M. Skapura. Neural Networks: Algo-rithms, Applications, and Programming Techniques. Addison Wesley,1991.[FvDFH90] James D. Foley, Andries van Dam, Steven K. Feiner, and John F.Hughes. Computer Graphics: Principles and Practice. Addison-Wesley,Reading, MA, 1990. Second Edition.[FW83]T. W. Finin and B. L. Webber. BUP { A Bottom Up Parser. Techni-cal Report MS-CIS-83-16, Computer and Information Science, Univ. ofPennsylvania, Philadelph ia, PA, 1983.[FW88]David R. Forsey and Jane Wilhelms. Techniques for interactive manip-ulation of articulated bodies using dynamic analysis. In Proceedings ofGraphics Interface ’88, 1988.[Gal80]C. R. Gallistel. The Orginization of Action: A New Synthesis. LawrenceElerbaum Associates, Publishers, Hillsdale, NJ, 1980. Distributed bythe Halsted Press division of John Wiley & Sons.[Gan85]Je(cid:11)rey S. Gangel. A motion verb interface to a task animation system.Master’s thesis, Computer and Information Science, Univ. of Pennsyl-vania, Philadelphia, PA, August 1985.[Gei92]Christopher Geib. Intentions in means-end planning. Technical ReportMS-CIS-92-73, Dept. of Computer and Information Science, Univ. ofPennsylvania, Philadelphia, PA, 1992.[GFS71]R. M. Goldwyn, H. P. Friedman, and T. H. Siegel. Iteration and interac-tion in computer data bank analysis. Computer in Biomedical Research,[Gir87]Michael Girard.Interactive design of 3D computer-animated leggedanimal motion. IEEE Computer Graphics and Applications, 7(6):39{4:607{622, 1971.51, 1987.[Gir91]Michael Girard. Constrained optimization of articulated animal move-ment in computer animation. In Norman I. Badler, Brian A. Barsky,and David Zeltzer, editors, Making Them Move: Mechanics, Con-trol, and Animation of Articulated Figures, pages 209{232. Morgan-Kaufmann, San Mateo, CA, 1991.[GL90]Michael P. George(cid:11) and Amy L. Lansky. Reactive reasoning and plan-ning. In James Allen, James Hendler, and Austin Tate, editors, Read-ings in Planning, pages 729{734. Morgon Kaufmann Publishers, Inc.,[GM85]Michael Girard and A. A. Maciejewski. Computational modeling for thecomputer animation of legged (cid:12)gures. Computer Graphics, 19(3):263{1990.270, 1985.[GM86]Carol M. Ginsberg and Delle Maxwell. Graphical marionette. In N. I.Badler and J. K. Tsotsos, editors, Motion: Representation and Percep-tion, pages 303{310. Elsevier, North Holland, New York, NY, 1986.[GMTT89] Jean-Paul Gourret, Nadia Magnenat-Thalmann, and Daniel Thalmann.Simulation of ob ject and human skin deformations in a grasping task.BIBLIOGRAPHY257Computer Graphics, 23(3):21{30, 1989.[Gol69]D. Goldfarb. Extension of Davidon’s variable metric method to maxi-mization under linear inequality and equality constraints. SIAM Jour-nal of Appl. Math., 17:739{764, 1969.[Gol70]D. Goldfarb. A family of variable metric methods derived by variationalmeans. Math. Computation, 24:23{26, 1970.[Gom84]Julian E. Gomez. Twixt: A 3D animation system. In Proc. Eurograph-ics ’84, pages 121{133, New York, NY, July 1984. Elsevier SciencePublishers B.V.[Gou84]Laurent Gouzenes. Strategies for solving collision-free tra jectories prob-lems for mobile and manipulator robots. Int. Journal of Robotics Re-[GP88]Ralph Guggenheim and PIXAR. Tin Toy (excerpt). SIGGRAPH Videosearch, 3(4):51{65, Winter 1984.Review, 38, 1988.[GQB89] Marc Grosso, Richard Quach, and Norman I. Badler. Anthropometryfor computer animated human (cid:12)gures. In N. Magnenat-Thalmann andD. Thalmann, editors, State-of-the Art in Computer Animation, pages83{96. Springer-Verlag, New York, NY, 1989.+[GQO89] Marc Grosso, Richard Quach, Ernest Otani, Jianmin Zhao, SusannaWei, Pei-Hwa Ho, Jiahe Lu, and Norman I. Badler. Anthropometryfor computer graphics human (cid:12)gures. Technical Report MS-CIS-89-71,Computer and Information Science, Univ. of Pennsylvania, Philadel-phia, PA, 1989.[GR82]K. Gupta and B. Roth. Design considerations for manipulatorworkspace. ASME Journal of Mechanical Design, 104:704{711, Oct.1982.+[GRB85] S. M. Goldwasser, R. A. Reynolds, T. Bapty, D. Bara(cid:11), J. Summers,D. A. Talton, and E. Walsh. Physician’s workstation with real-timeperformance. IEEE Computer Graphics and Applications, 5(12):44{57,[GS89]Barbara Grosz and Candice Sidner. Plans for discourse. In J. Morgan,P. Cohen, and M. Pollack, editors, Intentions in Communication. MITDec. 1985.Press, 1989.[Gup86]K. Gupta. On the nature of robot workspace. Int. Journal of RoboticsResearch, 5:112{122, 1986.[Gup90]Kamal Kant Gupta. Fast collision avoidance for manipulator arms: Asequential search strategy.IEEE Transactions on Robotics and Au-tomation, 6(5):522{532, Oct 1990.[Hac77]R. J. Hackathorn. ANIMA II: A 3-D color animation system. Computer[Hah88]James K. Hahn. Realistic animation of rigid bodies. Computer Graph-Graphics, 11(2):54{64, July 1977.ics, 22(4):299{308, August 1988.[Har75]J. A. Hartigan. Printer graphics for clustering. Journal of StatisticalComputation and Simulation, 4:187{213, 1975.[Hau89]Edward J. Haug, editor. Concurrent Engineering of Mechanical Sys-tems: Volume I. The Univ. of Iowa, Iowa City, IA, 1989.[HBD80]R. Harris, J. Bennet, and L. Dow. CAR-II { A revised model for crew as-sesment of reach. Technical Report 1400.06B, Analytics, Willow Grove,PA, 1980.[HC90]Adele E. Howe and Paul R. Cohen. Responding to environmental258BIBLIOGRAPHYchange. Proceedings of the ARPA Workshop on Planning, Scheduling,and Control, pages 85{92, Nov. 1990.[HE78]Don Herbison-Evans. NUDES2: A numeric utility displaying ellipsoidsolids. Computer Graphics, 12(3):354{356, Aug. 1978.[HE82]Don Herbison-Evans. Real-time animation of human (cid:12)gure drawingswith hidden lines omitted. IEEE Computer Graphics and Applications,2(9):27{34, 1982.[Her86]Annette Herskovits. Language and spatial cognition. In Aravind Joshi,editor, Studies in Natural Language Processing. Cambridge Univ. Press,Cambridge, England, 1986.[HH87]C. Ho(cid:11)mann and R. Hopcroft. Simulation of physical systems fromgeometric models.IEEE Journal of Robotics and Automation, RA-3(3):194{206, 1987.[Hir77]Vicki Hirsch. Floorplans in Labanotation. Master’s thesis, Computerand Information Science, Univ. of Pennsylvania, Philadelphia, PA,[HJER86] V. H. Heyward, S. M. Johannes-Ellis, and J. F. Romer. Gender di(cid:11)er-ences in strength. Research Quarterly for Exercise and Sport, 57(2):154{1977.159, 1986.phia, PA, 1982.4(6), 1988.[HKP91]John Hertz, Anders Krogh, and Richard G. Palmer. Introduction to thetheory of neural computation. Addison Wesley, 1991.[Hol81]W. H. Hollinshead. Functional Anatomy of the Limbs and Back. Saun-ders, Philadelphia, PA, 1981.[Hol82]W. H. Hollinshead. Anatomy for Surgeons. Harper & Row, Philadel-[HP88]David R. Haumann and Richard E. Parent. The behavioral testbed:Obtaining complex behavior from simple rules. The Visual Computer,[HS85a]Pat Hanrahan and David Sturman. Interactive animation of parametricmodels. The Visual Computer, 1(4):260{266, 1985.[HS85b]J. M. Hollerbach and K. C. Suh. Redundancy resolution of manipula-tors through torque optimization. In IEEE Intl. Conf. on Robotics andAutomation, pages 1016{1021, St. Louis, MO, 1985.[Hut70]Ann Hutchinson. Labanotation. Theatre Arts Books, New York, NY,[Hut84]Ann Hutchinson. Dance Notation. Dance Horizons, New York, NY,[Ibe87]T. Iberall. The nature of human prehension: Three dextrous hands inone. In IEEE Intl. Conf. on Robotics and Automation, pages 396{401,[IC87]Paul M. Isaacs and Michael F. Cohen. Controlling dynamic simulationwith kinematic constraints. Computer Graphics, 21(4):215{224, 1987.[Imr83]S. N. Imrhan. Model ling Isokinetic Strength of the Upper Extremity.PhD thesis, Texas Tech Univ., 1983.[IRT81]Verne T. Inman, Henry J. Ralston, and Frank Todd. Human Walking.Williams and Wilkins, Baltimore, MD, 1981.[Jac90]Ray Jackendo(cid:11). Semantic Structures. MIT Press, Cambridge, MA,1970.1984.1987.1990.[JCMM73] Reid Joyce, Andrew Chenzo(cid:11), Joseph Mulligan, and William Mallory.Fully proceduralized job performance aids. Technical Report AFHRL-BIBLIOGRAPHY259Tr-73-43(I), Air Force Human Resources Laboratory, Wright-PattersonAFB, 1973.[JKBC91] Moon Jung, Jugal Kalita, Norman I. Badler, and Wallace Ching. Simu-lating human tasks using simple natural language instructions. In Proc.Winter Simulation Conf., Phoenix, AZ, 1991.[JM85]R. J. Jagacinski and D. L. Monk. Fitts’ Law in two dimensions withhand and head movements. Journal of Motor Behavior, 17, 1985.[Joh76]G. Johansson. Spatial-temporal di(cid:11)erentiation and integration in visualmotion perception. Psychology Research, 38:379{383, 1976.[Jun92]Moon Jung. Human-Like Agents with Posture Planning Ability. PhDthesis, Computer and Information Science, Univ. of Pennsylvania,Philadelphia, PA, 1992.[Kae90]Leslie P. Kaelbling. An architecture for intelligent reactive systems.In James Allen, James Hendler, and Austin Tate, editors, Readings inPlanning, pages 713{728. Morgon Kaufmann Publishers, Inc., 1990.[Kal90]Jugal Kumar Kalita. Natural Language Control of Animation of TaskPerformance in a Physical Domain. PhD thesis, Computer and Infor-mation Science, Univ. of Pennsylvania, Philadelphia, PA, 1990.[Kar87]Robin Karlin. SEAFACT: A semantic analysis system for task anima-tion of cooking operations. Master’s thesis, Computer and InformationScience, Univ. of Pennsylvania, Philadelphia, PA, Dec. 1987.[Kar88]Robin Karlin. De(cid:12)ning the semantics of verbal modi(cid:12)ers in the domainof cooking tasks. In Proc. of the 26st Annual Meeting of ACL, pages[KB82]James U. Korein and Norman I. Badler. Techniques for goal directedmotion. IEEE Computer Graphics and Applications, 2(9):71{81, Nov.[KB90]Jugal Kalita and Norman I. Badler. Semantic analysis of a class ofaction verbs based on physical primitives. In Proc. 12th Annual Con-ference of the Cognitive Science Society, pages 412{419, Boston, MA,[KB91]Jugal Kalita and Norman I. Badler. Interpreting prepositions physically.In Proc. AAAI-91, pages 105{110, Anaheim, CA, 1991.[Kee82]Steve W. Keele. Learning and control of coordinated motor patterns:The programming perspective.In J.A. Scott Kelso, editor, HumanMotor Behavior. Lawrence Erlbaum Associates, 1982.[KH81]B. Kleiner and J. A. Hartigan. Representating points in many dimen-sions by trees and castles. Journal of American Statistical Association,76(374):260{269, 1981.[KH83]C.A. Klein and C.H. Huang. Review of pseudoinverse control for usewith kinematically redundant manipulators. IEEE Transactions on Sys-tems, Man and Cybernetics, 13(2), 1983.[Kha86]O. Khatib. Real-time obstacle avoidance for manipulators and mobilerobots. Int. Journal of Robotics Research, 5(1):90{98, 1986.[Kha87]O. Khatib. A uni(cid:12)ed approach for motion and force control of robotmanipulators: The operational space formulation.IEEE Journal ofRobotics and Automation, RA-3(1):43{53, 1987.[KKB88]Scott Kushnier, Jugal Kalita, and Norman I. Badler. Constraint-basedtemporal planning. Technical report, Computer and Information Sci-ence, Univ. of Pennsylvania, Philadelphia, PA, 1988.61{67, 1988.1982.July 1990.260BIBLIOGRAPHY[KN87]K. Kazerounian and A. Nedungadi. An alternative method for mini-mization of driving forces in redundant manipulators.In IEEE Intl.Conf. on Robotics and Automation, pages 1701{1706, Raleigh, NC,[Kor85]James U. Korein. A Geometric Investigation of Reach. MIT Press,1987.Cambridge, MA, 1985.[KR79]M. E. Kahn and B. Roth. The near-minimum time control of open looparticulated kinematic chains. Transactions of the ASME: Journal ofDynamic Systems, Measurement, and Control, 93(3):164{172, 1979.[KSC81]E. Kingsley, N. Scho(cid:12)eld, and K. Case. SAMMIE { A computer aidfor man-machine modeling. Computer Graphics, 15(3):163{169, Aug.1981.+[KTV90] James P. Karlen, Jack M Thompson, Havard I. Vold, James D. Far-rell, and Paul H. Eismann. A dual-arm dexterous manipulator systemwith anthropomorphic kinematics. In IEEE Intl. Conf. on Robotics and[Kum80]A. Kumar. Characterization of Manipulator Geometry. PhD thesis,Automation, 1990.Univ. of Houston, 1980.[KW81]A. Kumar and K. Waldron. The workspace of a mechanical manipula-tor. ASME Journal of Mechanical Design, 103:665{672, July 1981.[KZ86]Kamal Kant and Steven W. Zucker. Toward e(cid:14)cient tra jectory plan-ning: The path-velocity decomposition. Int. Journal of Robotics Re-search, 5(3):72{89, Fall 1986.[Lau76]L. L. Laubach. Comparative muscular strength of men and women: Areview of the literature. Aviation, Space, and Environmental Medicine,47(5):534{542, 1976.[LCF76]G. D. Langolf, D. B. Cha(cid:14)n, and J. A. Foulke. An investigation ofFitts’ Law using a wide range of movement amplitudes. Journal ofMotor Behavior, 8, 1976.[Lee92]Philip L. Y. Lee. Modeling Articulated Figure Motion with Physical ly-and Physiologically-Based Constraints. PhD thesis, Mechanical Engi-neering and Applied Mechanics, Univ. of Pennsylvania, Philadelphia,PA, 1992.[Lev77]Marc Levoy. A color animation system based on the multi-plane tech-nique. Computer Graphics, 11(2):64{71, July 1977.[Lev91]Libby Levison. Action composition for the animation of Natural Lan-guage instructions. Technical Report MS-CIS-91-28, Computer andInformation Science, Univ. of Pennsylvania, Philadelphia, PA, 1991.[Lif91]Kinetic E(cid:11)ects, Inc., Seattle, WA. Life Forms User Manual, 1991.[Loc91]K. Lochbaum. An algorithm for plan recognition in collaborative dis-thcourse. In Proc. 29Annual Meeting of the Assoc. for ComputationalLinguistics, pages 33{38, Berkeley, CA, June 1991.[Lou83]R. Louis. Surgery of the Spine. Springer-Verlag, New York, NY, 1983.[LP81]Tomas Lozano-P(cid:19)erez. Automatic planning of manipulator transfermovements.IEEE Transactions on Systems, Man and Cybernetics,SMC-11(10):681{698, Oct 1981.[LP83]Tomas Lozano-P(cid:19)erez. Spatial planning: A con(cid:12)guration space ap-proach. IEEE Transactions on Computers, c-32(2):26{37, Feb 1983.[LP87]Tomas Lozano-P(cid:19)erez. A simple motion planning algorithm for generalrobot manipulators. IEEE Journal of Robotics and Automation, RA-BIBLIOGRAPHY2613(3):224{238, June 1987.[LPW79]T. Lozano-P(cid:19)erez and M. A. Wesley. An algorithm for planning collision-free paths among polyhedral obstacles. Communications of the ACM,22(10):560{570, Oct. 1979.[LRM88]Timothy Lohman, Alex Roche, and Reynaldo Martorell. Anthropomet-ric Standardization Reference Manual. Human Kinetic Books, Cham-paign, IL, 1988.[LWZB90] Philip Lee, Susanna Wei, Jianmin Zhao, and Norman I. Badler.Strength guided motion. Computer Graphics, 24(4):253{262, 1990.[LY83]T. Lee and D. Yang. On the evaluation of manipulator workspace. Jour-nal of Mechanisms, Transmissions, and Automation in Design, 105:70{77, March 1983.[Mae90]Pattie Maes. Situated agents can have goals. In Pattie Maes, editor,Designing Autonomous Agents, pages 49{70. MIT Press, 1990.[Mau91]Ruth A. Maulucci. Personal communication, 1991.[MB77]M. A. MacConaill and J. V. Basma jian. Muscles and Movements, aBasic for Human Kinesiology. R. E. Krieger, Huntington, NY, 1977.[MB91]G. Monheit and N. Badler. A kinematic model of the human spine andtorso. IEEE Computer Graphics and Applications, 11(2):29{38, 1991.[McD89]J. W. McDaniel. Modeling strength data for CREW CHIEF. In Pro-ceedings of the SOAR 89 (Space Operations, Automation, and Robotics),Johnson Space Center, Houston, TX, July 1989.[Mil88]Gavin S. P. Miller. The motion dynamics of snakes and worms. Com-puter Graphics, 22(4):169{178, 1988.[Mil91]Gavin Miller. Goal-directed animation of tubular articulated (cid:12)guresor how snakes play golf. In Norman I. Badler, Brian A. Barsky, andDavid Zeltzer, editors, Making Them Move: Mechanics, Control, andAnimation of Articulated Figures, pages 209{233. Morgan-Kaufmann,San Mateo, CA, 1991.[Min86]Marvin Minsky. The Society of Mind. Simon and Schuster, 1986.[MK85]A. A. Maciejewski and C. A. Klein. Obstacle avoidance for kinemat-ically redundant manipulators in dynamically varying environments.Int. Journal of Robotics Research, 4(3):109{117, 1985.+[MKK88] J. McDaniel, M. Korna, P. Krauskopf, D. Haddox, S. Hardyal, M. Jones,and J. Polzinetti. User’s Guide for CREW CHIEF: A computer graph-ics simulation of an aircraft maintenance technician. Technical report,Armstrong Aerospace Medical Research Laboratory, Human SystemsDivision, Air Force System Command, Wright-Patterson Air ForceBase, OH, May 1988.[MPZ90] Michael McKenna, Steve Pieper, and David Zeltzer. Control of a virtualactor: The roach. Computer Graphics, 24(2):165{174, 1990.[MS86]A. Mital and N. Sanghavi. Comparison of maximum volitional torqueexertion capabilities of males and females using common hand tools.Human Factors, 28(3):283{294, 1986.[MTT85] Nadia Magnenat-Thalmann and Daniel Thalmann. Computer Anima-tion: Theory and Practice. Springer-Verlag, New York, NY, 1985.[MTT90] Nadia Magnenat-Thalmann and Daniel Thalmann. Synthetic Actors in3-D Computer-Generated Films. Springer-Verlag, New York, NY, 1990.[MTT91a] Nadia Magnenat-Thalmann and Daniel Thalmann. Complex models foranimating synthetic actors. IEEE Computer Graphics and Applications,262BIBLIOGRAPHY11(5):32{44, Sept. 1991.[MTT91b] Nadia Magnenat-Thalmann and Daniel Thalmann. Human body de-formations using joint-dependent local operators and (cid:12)nite-element the-ory. In Norman I. Badler, Brian A. Barsky, and David Zeltzer, editors,Making Them Move: Mechanics, Control, and Animation of ArticulatedFigures, pages 243{262. Morgan-Kaufmann, San Mateo, CA, 1991.[Muj87]C. Mujabbir. Workspaces of serial manipulators. Master’s thesis, Me-chanical Engineering and Applied Mechanics, Univ. of Pennsylvania,[NAS78]NASA. The Anthropometry Source Book. NASA Reference Publication1024, Johnson Space Center, Houston, TX, 1978. (Two volumes).[NAS87]NASA. Man-System Integration Standards. NASA-STD-3000, March1987.1987.[Nel85]Greg Nelson. Juno, a constraint-based graphics system. ComputerGraphics, 19(3):235{243, 1985.+[NHK85] H. Nishimura, M. Hirai, T. Kawai, T. Kawata, I. Shirakawa, andK. Omura. Ob ject modeling by distribution function and a methodof image generation. In Proc. Electronics Communication Conf., vol-ume J68-D(4), 1985. (in Japanese).[NHK86]NHK. Caron’s world. SIGGRAPH Video Review, 24, 1986.[NO87]Gregory Nielson and Dan Olsen Jr. Direct manipulation techniquesfor 3D ob jects using 2D locator devices. In Proceedings of 1986 ACMWorkshop on Interactive 3D Graphics, Chapel Hill, NC, Oct. 1987.[OB80]Joseph O’Rourke and Norman I. Badler. Model-based image analysis ofhuman motion using constraint propagation. IEEE Trans. on PatternAnalysis and Machine Intel ligence, 2(6):522{536, Nov. 1980.[OO81]T. J. O’Donnell and Arthur J. Olson. GRAMPS { A graphics languageinterpreter for real-time, interactive, three-dimensional picture editingand animation. Computer Graphics, 15(3):133{142, 1981.[Ota89]Ernest Otani. Software tools for dynamic and kinematic modeling ofhuman motion. Technical Report MS-CIS-89-43, Computer and Infor-mation Science, Univ. of Pennsylvania, Philadelphia, PA, 1989. (MSEThesis, Mechanical Engineering and Applied Mechanics, Univ. of Penn-sylvania).[Pau81]Richard Paul. Robot Manipulators: Mathematics, Programming, andControl. MIT Press, Cambridge, MA, 1981.[PB88]Cary Phillips and Norman I. Badler. Jack: A toolkit for manipulatingarticulated (cid:12)gures. In Proceedings of ACM SIGGRAPH Symposium onUser Interface Software, pages 221{229, Ban(cid:11), Canada, Oct. 1988.[PB91]Cary B. Phillips and Norman I. Badler. Interactive behaviors for bipedalarticulated (cid:12)gures. Computer Graphics, 25(4):359{362, 1991.[PBS91]Catherine Pelachaud, Norman I. Badler, and Mark Steedman. Issuesin facial animation. In Computer Animation ’91, Geneva, Switzerland,1991.[Pel91]Catherine Pelachaud. Communication and coarticulation in facial ani-mation. PhD thesis, Computer and Information Science, Univ. of Penn-sylvania, Philadelphia, PA, 1991. Tech. Report MS-CIS-91-82.[Pen86]Alex Pentland. Perceptual organization and the representation of nat-ural form. AI Journal, 28(2):1{38, 1986.+[PMA91] A. Pandya, J. Maida, A. Aldridge, S. Hasson, and B. Woolford. Devel-BIBLIOGRAPHY263opment of an empirically based dynamic biomechanical strength model.In Space Operations Applications and Research Conf. Proc., pages 438{[Pot91]Caren Potter. The human factor. Computer Graphics World, pages444, 1991. Vol. 2.61{68, March 1991.[Pow70]M. J. D. Powell. A hybrid method for nonlinear equations. In P. Rabi-nowitz, editor, Numerical Methods for Nonlinear Algebraic Equations.Gordon and Breach Science, 1970.[PR90]Martha E. Pollack and Marc Ringuette.Introducing Tileworld: Ex-perimentally evaluating an agent architecture. Proceedings of the 8thNational Conference on Arti(cid:12)cial Intel ligence, pages 183{189, 1990.[Pra84]P. Prasad. An overview of ma jor occupant simulation models. In Pro-ceedings of Society of Automotive Engineers, 1984. Paper No. 840855.[Pri81]Ellen Prince. Toward a taxonomy of given/new information. In P. Cole,editor, Radical Pragmatics, pages 223{255. Academic Press, New York,NY, 1981.[PW89]A. Pentland and J. Williams. Good vibrations: Modal dynamics forgraphics and animation. Computer Graphics, 23(3):215{222, 1989.[PZB90]Cary Phillips, Jianmin Zhao, and Norman I. Badler. Interactive real-time articulated (cid:12)gure manipulation using multiple kinematic con-straints. Computer Graphics, 24(2):245{250, 1990.[RA90]David F. Rogers and J. Alan Adams. Mathematical Elements for Com-puter Graphics. McGraw-Hill, New York, NY, 1990. Second Ed.[Ree83]William T. Reeves. Particle systems { A technique for modelling a classof fuzzy ob jects. Computer Graphics, 17(3):359{376, July 1983.[Rey82]Craig W. Reynolds. Computer animation with scripts and actors. Com-puter Graphics, 16(3):289{296, July 1982.[Rey87]Craig W. Reynolds. Flocks, herds, and schools: A distributed behav-ioral model. Computer Graphics, 21(4):25{34, 1987.[Rey88]Craig W. Reynolds. Not bumping into things. SIGGRAPH course 27notes: Developements in Physically-Based Modeling, 1988. G1{G13.[RG91]Hans Rijpkema and Michael Girard. Computer animation of hands andgrasping. Computer Graphics, 25(4):339{348, July 1991.[RMTT90] Olivier Renault, Nadia Magnenat-Thalmann, and Daniel Thalmann. Avision-based approach to behavioral animation. The Journal of Visual-ization and Computer Animation, 1(1):18{21, 1990.[Ros60]J. B. Rosen. The gradient pro jection method for nonlinear programming{ I: Linear constraints. SIAM Journal of Appl. Math., 8:181{217, 1960.[Ros91]David A. Rosenbaum. Human Motor Control. Academic Press, 1991.[Rot75]B. Roth. Performance evaluation of manipulators from a kinematicsviewpoint. NBS Special Publication, pages 39{61, 1975.[SB85]Scott Steketee and Norman I. Badler. Parametric keyframe interpola-tion incorporating kinetic adjustment and phrasing control. ComputerGraphics, 19(3):255{262, 1985.[Sch72]F. T. Schanne. Three Dimensional Hand Force Capability Model for aSeated Person. PhD thesis, Univ. of Michigan, Ann Arbor, MI, 1972.[Sch82a]Richard Schmidt. More on motor programs. In J.A. Scott Kelso, editor,Human Motor Behavior. Lawrence Erlbaum Associates, 1982.[Sch82b]Richard Schmidt. The schema concept.In J.A. Scott Kelso, editor,Human Motor Behavior. Lawrence Erlbaum Associates, 1982.264BIBLIOGRAPHY1992.1981.1987.1986.[Sch83]Christopher Schmandt.Spatialinput/display correspondence in astereoscopic computer graphics workstation.Computer Graphics,17(3):253{261, July 1983.[Sch90]J. H. Schmidhuber. Making the world di(cid:11)erentiable: On using su-pervised learning fully recurrent networks for dynamic reinforcementlearning and planning in non-stationary environments. Technical Re-port FKI-126-90, Technische Universit(cid:127)at M(cid:127)unchen, Febuary 1990.[SEL84]SELF. The (cid:12)rst 3-D computer exercises, Sept. 1984.[SH86]G. Sahar and J. M. Hollerbach. Planning of minimum-time tra jactoriesfor robot arms. Int. Journal of Robotics Research, 5(3):90{100, 1986.[Sha70]D. F. Shanno. Conditioning of quasi-Newton methods for function min-imization. Math. Computation, 24:647{664, 1970.[Sha80]U. Shani. Filling regions in binary raster images: A graph-theoreticapproach. Computer Graphics, 14(3):321{327, 1980.[Sha88]Lokendra Shastri. A connectionist approach to knowledge representa-tion and limited inference. Cognitive Science, 12(3):331{392, 1988.[Sho92]Ken Shoemake. ARCBALL: A user interface for specifying three-dimensional orientation using a mouse. In Proceedings of SIGCHI ’92,[Sim81]Herbert A. Simon. The Sciences of the Arti(cid:12)cial. MIT Press, 2 edition,[SL87]S. Singh and M. C. Leu. Optimal tra jectory generation for roboticmanipulators using dynamic programming. Transactions of the ASME:Journal of Dynamic Systems, Measurement, and Control, 109:88{96,[SP86]Thomas W. Sederberg and Scott R. Parry. Free-form deformation ofsolid geometric models. Computer Graphics, 20(4):151{160, August[SS83a]J. T. Schwartz and M. Sharir. On the piano movers’ problem { I: Thecase of a two dimensional rigid polygonal body moving amidst polygonalbarriers. Communications on Pure and Applied Mathematics, 36:345{398, 1983.[SS83b]J. T. Schwartz and M. Sharir. On the piano movers’ problem { II: Gen-eral techniques for computing topological properties of real algebraicmanifolds. Advances in Applied Mathematics, 4:298{351, 1983.[SSSN85] D. Schmitt, A. H. Soni, V. Srinivasan, and G. Naganthan. Optimal mo-tion programming of robot manipulators. Transactions of the ASME:Journal of Mechanisms, Transmissions, and Automation in Design,[Ste83]G. Stern. BBOP { A program for 3-Dimensional animation. In Nico-107:239{244, 1985.graph ’83, Tokyo, Japan, 1983.Philosophy, 13:207{263, 1990.[Ste90]Mark Steedman. Gapping as constituent coordination. Linguistics and[Stu84]David Sturman.Interactive key frame animation of 3-D articulatedmodels. In Proc. Graphics Interface ’84, pages 35{40, Ottawa, Canada,1984.[Sug81]D. Sugimoto. Determination of extreme distances of a robot hand.ASME Journal of Mechanical Design, 103:631{636, July 1981.[Sun91]Ron Sun. Neural network models for rule-based reasoning. In IEEE In-ternational Joint Conference on Neural Networks, pages 503{508, Sin-BIBLIOGRAPHY265gapore, 1991.[TA75]G. J. Torotra and N. P. Anagnostakos. Principles of Anatomy andPhysiology. Can(cid:12)eld Press, New York, NY, 1975.[TJ81]Frank Thomas and Ollie Johnson. Disney Animation: The Il lusion ofLife. Abbeville Press, New York, NY, 1981.[TPBF87] Demetri Terzopoulos, John Platt, Alan Barr, and Kurt Fleischer. Elas-tically deformable models. Computer Graphics, 21:205{214, 1987.[TS81]Y. Tsai and A. Soni. Accessible region and synthesis of robot arms.ASME Journal of Mechanical Design, 103:803{811, Oct. 1981.[TS83]Y. Tsai and A. Soni. An algorithm for the workspace of a general n-Rrobot. ASME Journal of Mechanical Design, 105:52{57, July 1983.[Tsa86]M. Tsai. Workspace Geometric Characterization and Manipulability ofIndustrial Robots. PhD thesis, Ohio State Univ., 1986.[TST87]Yosuke Takashima, Hideo Shimazu, and Masahiro Tomono.Storydriven animation. In CHI + GI ’87 Proceedings, pages 149{153. ACMSIGCHI, 1987.[Tur63]A. M. Turing. Computing machinery and intelligence. In E. A. Feigen-baum and J. Feldman, editors, Computers and Thought, pages 11{35.McGraw-Hill, New York, NY, 1963.[VB90]S. A. Vere and T. W. Bickmore. A basic agent. Computational Intel li-[Vij85]R. Vijaykumar. Robot manipulators { Workspaces and geometricaldexterity. Master’s thesis, Mechanical Engineering, Ohio State Univ.,gence, 6:41{60, 1990.Columbus, OH, 1985.[Wav89] Wavefront Technologies. MODEL User’s Manual Version 6.0, 1989.[WB85]Jane Wilhelms and Brian A. Barsky. Using dynamics for the animationof articulated bodies such as humans and robots.In Proc. GraphicsInterface ’85, pages 97{104, Montreal, Canada, 1985.[WB92]Susanna Wei and Norman I. Badler. Graphical displays of humanstrength data. Visualization and Computer Animation, 3(1):13{22,[Wei86]Jerry Weil. The synthesis of cloth ob jects. Computer Graphics,1992.20(4):49{54, 1986.[Wei90]Susanna Wei. Human Strength Database and Multidimensional DataDisplay. PhD thesis, Computer and Information Science, Univ. of Penn-[Wel71]K. Wells. Kinesiology, the Scienti(cid:12)c Basis of Human Action. Saunders,sylvania, Philadelphia, PA, 1990.Philadelphia, PA, 1971.[Wes73]Barry D. Wessler. Computer-assisted visual communication. PhD thesis,Univ. of Utah, Salt Lake City, UT, 1973.[WFB87] Andrew Witkin, Kurt Fleisher, and Alan Barr. Energy constraints onparameterized models. Computer Graphics, 21(3):225{232, 1987.[Whi72]D. E. Whitney. The mathematics of coordinated control of prosthesesand manipulators. J. Dynamic Systems, Measurement, and Control,Transaction ASME, 94:303{309, 1972. Series G.[Wil75]F. Wilson, editor. The Musculoskeletal System. Lippincott, Philadel-phia, PA, 1975.[Wil82]K. D. Willmert. Visualizing human body motion simulations. IEEEComputer Graphics and Applications, 2(9):35{38, Nov. 1982.[Wil86]Jane Wilhelms. Virya { A motion editor for kinematic and dynamic266BIBLIOGRAPHYanimation. In Proc. Graphics Interface ’86, pages 141{146, Vancouver,[Wil87]Jane Wilhelms. Using dynamic analysis for realistic animation of artic-ulated bodies. IEEE Computer Graphics and Applications, 7(6):12{27,Canada, 1986.1987.[Wil91]Jane Wilhelms. Dynamic experiences. In Norman I. Badler, Brian A.Barsky, and David Zeltzer, editors, Making Them Move: Mechanics,Control, and Animation of Articulated Figures, pages 265{279. Morgan-Kaufmann, San Mateo, CA, 1991.[Win90]David A. Winter. Biomechanics and Motor Control of Human Move-ment. Wiley Interscience, New York, NY, 1990. Second Edition.[WK88]A. Witkin and M. Kass. Spacetime constraints. Computer Graphics,22(4):159{168, 1988.[WMW86] G. Wyvill, C. McPheeters, and B. Wyvill. Data structure for soft ob-jects. The Visual Computer, 2:227{234, 1986.[WS90]Jane Wilhelms and Robert Skinner. A ‘notion’ for interactive behav-ioral animation control.IEEE Computer Graphics and Applications,10(3):14{22, May 1990.[WSB78]Lynne Weber, Stephen W. Smoliar, and Norman I. Badler. An archi-tecture for the simulation of human movement. In Proc. ACM AnnualConf., pages 737{745, Washington, DC, 1978.[WW90]Andrew Witkin and William Welch. Fast animation and control ofnonrigid structures. Computer Graphics, 24(4):243{252, 1990.[Yeo76]B. P. Yeo.Investigations concerning the principle of minimal totalmuscular force. Journal of Biomechanics, 9:413{416, 1976.[YL83]D. Yang and T. Lee. On the workspace of mechanical manipulators.Journal of Mechanisms, Transmissions, and Automation in Design,105:62{69, March 1983.[YN87]V. Yen and M. L. Nagurka. Suboptimal tra jectory planning of a (cid:12)ve-linkhuman locomotion model. In Biomechanics Proceedings, 1987.[ZB89]Jianmin Zhao and Norman I. Badler. Real time inverse kinematics withjoint limits and spatial constraints. Technical Report MS-CIS-89-09,Computer and Information Science, Univ. of Pennsylvania, Philadel-phia, PA, 1989.[Zel82]David Zeltzer. Motor control techniques for (cid:12)gure animation. IEEEComputer Graphics and Applications, 2(9):53{59, Nov. 1982.[Zel84]David Zeltzer. Representation and Control of Three Dimensional Com-puter Animated Figures. PhD thesis, The Ohio State Univ., 1984.[Zel91]David Zeltzer. Task-level graphical simulation: Abstraction, represen-tation, and control. In Norman I. Badler, Brian A. Barsky, and DavidZeltzer, editors, Making Them Move: Mechanics, Control, and Anima-tion of Articulated Figures, pages 3{33. Morgan-Kaufmann, San Mateo,CA, 1991.[ZV92]J. Zwarts and H. Verkuyl. An algebra of conceptual structure: Aninvestigation into Jackendo(cid:11) ’s conceptual semantics. Linguistics andPhilosophy, 1992. To appear.IndexA* search 181, 183legs 123, 135, 192animal sciences 137, 145neck 106, 133animation window 126, 129palms 114, 133animation 19, 30pelvis 103, 105, 106, 109, 111,anthropometry 14, 49, 209113, 117, 130, 134, 196apertures 190shoulder 12, 39, 123, 184arti(cid:12)cial intelligence 1, 233skeleton 12, 28attachment 132skin 15, 24backtracking 184, 186, 191spine 35, 106, 120, 134balance line 110, 113, 117thigh 152balance 13, 78, 79, 103, 108, 110, 115,toes 103, 131117, 123, 129, 135, 170, 192,torso 12, 28, 35, 79, 103, 105,197, 221, 231109. 112, 122, 130, 132banking 154, 157waist 106, 132, 133, 152, 154behaviors 13, 16, 101, 102, 116, 127,wrists 123129, 135, 137, 139, 148, 154,boundary detection 98208, 214, 240, 245center of mass 13, 78, 79, 103, 105,binocular vision 17109, 114, 118, 123, 129, 131,biostereometric bodies 15, 27, 47, 59,135, 154, 15565checklists 209body partsclimbing 191ankle 151clothing 15, 45, 245arms 12, 40, 108, 114, 123, 132,cognition 139135, 182collision avoidance 86, 93, 141, 149,calf 152180, 181, 192, 195, 245clavicle 12, 39collision detection 15, 17, 46, 132, 143,elbows 80, 103, 109146, 195, 221eyes 17, 103, 106, 123comfort 13, 161, 163, 164, 173, 189face 15, 247added joint 169feet 12, 79, 103, 105, 111(cid:11)., 115,available torque 167, 189, 190117, 118, 129(cid:11)., 135, 151,jerk 169196pull back 168, 190(cid:12)ngers 34, 132recoil 169forearm 34reducing moment 168, 190hands 12, 13, 107, 114, 126, 132,communication 144, 146, 233, 247133, 135, 196complexity 191head 106, 123, 196, 213compliant motion 132heels 103, 130, 131, 151computer animation 75, 83, 126, 137,hips 123, 130, 152, 154180, 212, 222knees 103, 109, 152computer hardware 243267268INDEXcomputer-aided design 15, 94, 246ellipsoids 26conceptual structures 233oct-trees 25, 181Concurrent Engineering 4polygons 24con(cid:12)guration space 180potential functions 27con(cid:13)ict resolution 146soft models 27connectionism 138, 145spheres 26constraints 9, 14, 16, 75, 127, 224superquadrics 26container 218surface points 23contour tracing 98volumetric representation 23control points 195voxels 25, 97control theory 161globographic limits 12, 41control vectors 195grammar 233coordinated motion 174grasping 108, 126, 132crash simulation 7, 161grip 13Cspace groups 181handles 246curved path 151, 154human factors analysis 3, 4, 16, 46,damping 14349, 60, 207, 228dance notation 8human motion analysis 7direct manipulation 67, 80, 83, 147,human performance 10, 163, 208, 212,103214, 245discomfort 164, 167human population data 14, 49, 58DOF 12icons 71, 82, 128double stance 155, 158individual database 54, 56draping 48instructions 207, 222, 232, 244, 246,dynamics 10, 53, 85, 143, 146, 152,247166intention 235, 246easing 128interactive modes 70e(cid:11)ectors 137interactive systems 10, 18, 30, 54, 70,E(cid:11)ort-Shape 9103, 145elasticity 144inverse kinematics 9, 17, 32, 75, 80,energy constraints 76, 77, 8583, 101, 108, 166, 170energy 173, 190jacks 69Eshkol-Wachmann 9Jacobian matrix 85, 166evolution 145joint angles 34, 83, 187, 189expert systems 137joint centers 28external forces 161joint gaps 46fatigue 173, 190joint group 34feedback 138, 141joint limits 12, 17, 37, 46, 52, 74, 84,(cid:12)eld of view 14987, 94, 99, 245Fitts’ Law 212, 231joint sti(cid:11)ness 93(cid:13)exible (cid:12)gures 35joints 12, 28, 30, 31, 51, 76, 78force tra jectory 163key pose 211free space 183kinesthetic feedback 67gait 101knowledge base 210, 224, 235gaze 135Labanotation 9, 102, 103, 105, 109,geometric models 16112(cid:11)., 117boundary representation 23learning 138, 245constructive solid geometry 26lexicon 219curved surfaces 24lifting 161, 170, 189cylinders 26load 163INDEX269local minima 48, 78, 80, 144across 218locomotion 146, 150in 218mass 14, 53, 57on 219mental simulation 195, 245pseudo-inverse 84message passing 147psychology 145minimum disturbance 199qualitative reasoning 192missing information 235quaternions 69mobile robot 182, 186radiosity 18moment of inertia 53randomness 144motion dependencies 199rate control 164, 166motion planning 180ray tracing 18motion postulator 199reachable space 18, 94motion understanding 8reactive planning 137, 142, 146motion 126, 128, 164, 182, 189, 197reasoning 192mouse line 72reference grounding 235multi-dimensional displays 61reference resolution 235, 236multiple agents 143, 174, 233, 247re(cid:13)exive behavior 138multiple constraints 83, 91, 126, 197,region (cid:12)lling 97215region graph 184multiple (cid:12)gures 13retinal pro jection 17names 210robotics 29, 75, 84, 94, 137, 163, 180,natural language 8, 10, 207, 209, 214,190222, 246roll 217networking 18root 31, 78, 79, 87, 92, 108, 112, 114,neural nets 138, 148118, 158neuro-ethology 138rotation transformation 68, 74nonlinear programming 75, 87, 96rotation turntable 68non-monotonic reasoning 145rotoscoping 2, 150ob ject references 210scaling rules 57, 59open 217script 18, 30, 209, 226, 228optimization 75, 78, 161, 165, 184,segments 28, 30, 46, 51, 58189, 192semantic representation 214paint system 211sensors 132, 137parallel systems 144shelves 190parser 219, 233, 235shoulder complex motion 41path planning 164, 166, 180, 190, 197simulation 10, 19, 60, 139, 141, 143,path trace 62, 94147, 152, 154, 197, 197, 207,perceived exertion 164221, 235perception 137, 139, 149, 238singularities 42physically-based models 30, 48, 141,site 13, 17, 31(cid:11), 44, 46, 107, 126, 132,142, 161135, 154, 209, 239plan expansion 235, 237base site 196plan graph 235situated action 137plan inference 235skill 246planning 137(cid:11)., 144, 163, 214, 221,skitters 69233, 237, 245speech 247postural control 75splines 147, 155posture planning 192, 245spreadsheet 14, 54potential (cid:12)eld 180stance leg 151prepositions 215stature 14, 57270INDEXstep length 152stepping 103, 115, 115, 118, 131, 135,150, 182, 192strength bar display 63strength box display 62180, 189, 245subsumption 137strength 13, 17, 54, 60, 61, 161, 164,support polygon 79, 109, 110, 115,117support 131, 221swing leg 151symbolic reasoning 137, 138task actions 224, 238task animation 10tasks 207, 222temporal planning 10texture map 211time line 127time 126, 129, 147, 182, 212, 230torques 17, 60, 62, 63, 65, 66, 153,translation transformation 68, 72161, 164tracks 126triangulation 98Turing Test 1turning 150, 154turntable 74verbs 214view cones 17view 17virtual humans 135, 244virtual reality 246virtual sphere 69virtual worlds 18visibility graph 184vision 141117, 131work 173, 190workspace 94, 150zero gravity 114volume visualization 96weight shift 79, 103, 110, 112, 113,Norman I. Badler is the Cecilia Fitler Moore Professor and Chair of Com-puter and Information Science at the University of Pennsylvania and hasbeen on that faculty since 1974. Active in computer graphics since 1968,his research focuses on human (cid:12)gure modeling, manipulation, and animation.Badler received the BA degree in Creative Studies Mathematics from the Uni-versity of California at Santa Barbara in 1970, the MSc in Mathematics in1971, and the Ph.D. in Computer Science in 1975, both from the Universityof Toronto. He is Co-Editor of the Journal Graphical Models and Image Pro-cessing. He also directs the Computer Graphics Research Laboratory withtwo full time sta(cid:11) members and about 40 students.Cary B. Phillips received his PhD in Computer and Information Sciencefrom the University of Pennsylvania in 1991, where he was a member of theTechnical Sta(cid:11) in the Computer Graphics Research Lab from 1988-91. He isthe principal designer and implementor of Jack. He received his BES and MSEdegrees in Electrical Engineering and Computer Science from The Johns Hop-kins University in 1985. Phillips is currently on the R&D sta(cid:11) at Paci(cid:12)c DataImages, a computer animation production company in Sunnyvale, California.Bonnie Lynn Webber is a Professor of Computer and Information Scienceat the University of Pennsylvania. She is known for her work in Computa-tional Linguistics { in particular, discourse processing and human-computerinteraction. She did her undergraduate work at MIT and her graduate workat Harvard, receiving a PhD in 1978. While at Harvard, she was also aSenior Scientist at Bolt Beranek and Newman Inc, Cambridge MA, whereshe contributed to the (cid:12)rst DARPA Speech Understanding Initiative. Shehas co-editted several books, including Readings in Arti(cid:12)cial Intel ligence andReadings in Natural Language Processing.PLATES1. The Jack (cid:12)gure inside a Deere and Company o(cid:11)-road vehicle. Vehiclecourtesy Jerry Duncan. Image by Cary Phillips and John Granieri.2. Apache helicopter cockpit with simulated operator. The helicopter cock-pit and helmet models were supplied by Barry Smith of thepro ject3AIat NASA Ames Research Center.Image by Pei-Hwa Ho, EunyoungKoh, Jiahe Lu, Welton Becket, Catherine Pelachaud, Soetjianto, andCary Phillips.and Eunyoung Koh.3. Graduation Day. Clothes modeling by Lauren Bello, Welton Becket,4. Translucent workspace for left hand reach. Image by Tarek Alameldin.Apache helicopter model courtesy Barry Smith at NASA Ames ResearchCenter.5. What you see is what you can reach with the left hand. Image by TarekAlameldin. Apache helicopter model courtesy Barry Smith at NASAAmes Research Center.6. \Go into the kitchen and get me the co(cid:11)ee urn." Scene from the \ani-mation from instructions" (AnimNL) pro ject. Image by Leanne Hwang,David Haynes, Brian Stokes, Moon Jung, and Aaron Fuegi.