Marc Breisinger Projects


LEMMA - Learning Environment with Multi Media Augmentations

LEMMA on the FogScreen


Mission Statement

This system has the purpose of exploiting the possibilities of a computer (namely visualisation, durability, multimodality etc.) to teach a student a portion of knowledge. This approach could be mapped to almost any kind of knowledge. It shall be explored to which extend this is true and how to develop teaching techniques for this approach.


The student shall have the experience to have a private teacher that goes at his/her own pace and provides exactly the amount of knowledge the student needs.

What is it gonna be?

A mixed reality environment for interactive computer tutorials. The first topic will be Physcis, namely Rigid Body Rotation

Requirements analysis

The student will:
  • read AND hear the lecturing, instructions, questions and feedback
  • see visualizations (if applicable) in 3D (Stereo) "2⅔D" (correct 3D perspective through head tracking) or 2½ D (3D graphics on 2D screen);example: in a physics tutorial we can have solid objects and representations of forces as vectors etc.
  • be able to interact with parts of the system, encouraging experimentation and promoting understanding. This is done mainly via hand-held input devices (dataglove, wand)
  • respond to inquiries by the system for the purpose of training, feedback/hints and evaluation
  • navigate linearly through a tutorial, following natural sequential flow of information (backward and forward)
  • navigate non-linearly through the tutorial, referring to recent and related content, using hyperlinks, bookmarks, etc.

The teacher will:
  • design multimedia tutorials, assisted by a multimedia authoring environment, In particular, the system provides functionality for
    • script structuring
    • text & voice synchronization
    • turorial flow control (pause, goto, wait for xy etc.)
    • visualiziation / illustration scheduling
    • various tools for visualiziation manipulation (highlighting, transparency etc.)
  • therefore be able to iterate and evolve the tutorial

System specification

The tutorial starts with a short introduction to itself, tells the student what is going to happen and how she can interact with the system. During the tutorial (nonregarding the content) the following things happen:
  • the lecturing text is rendered visualy and aurally
  • visualizations are shown and the student is asked to interact with it
  • if valueable formulas are shown next to the visualization in both abstract and concrete form with color coding (eg. t = r x F and (1,2,3) = (4,5,6) x (7,8,9) with the numbers updating in realtime)
  • the student reviews former taught knowledge and goes back to the current position
  • during and after the introduction the student will be inquiried about the topic (multiple choice, exercise e.g. apply right hand rule...). If possible when the student makes a mistake, the answer shall not just be "no" but a hint what could be different and maybe another question. (The questions should vary and be logged for not providing the same question twice).
  • alternation between instruction / exercise / inquiry

After the introduction follows the actual tutorial. So far the following tutorials have been written and deployed:


In a summarization the system will layout the hierarchy of the knowledge to make memorization easier to the student. At the end of the tutorial the student will be told about the achievements she has made and encouraged to go back to any of the subtutorials if anything remains unclear.


The System is build with Java and JavaOpenSceneGraph

Implementation design

System Architecture

System Architecture

Class Diagramm

Physics Engine

A rigid body is composed of Particles (atomic Rigid Bodies) and Parts (made of other Rigid Bodies)

Here you can find out how the engine works and what properties can be used in tutorials with this module.

Interactable GUIs

The user can interact with the interface of the learning application through interactable items, that support 2D and 3D manipulation. These can also be positioned through the modeled visualization.

Here is a detailed description which interactables are available and how the software architecture is designed.

The interactable items use partly common techniques for 3D interaction (like HOMER) but also combine or redefine techniques. This video shows the interaction techniques used in LEMMA.

Content Generation

A convenient and fast way of generating knowledge content is important as teachers should not have to struggle with programming but focus on the content they want to transfer. Furthermore in this concept the content should be able to evolve and therefore changes should be made easy. For this purpose LEMMA provides an authoring environment that gives the teacher the ability to write the "script" and control visualizations and other "real time" events that should occur during the tutorial. For generating 3D content a modeler like 3ds max can be used to produce visualizations that can be integrated in the LEMMA tutorials by just following a couple of conventions while modeling.

Authoring Environment

To author a tutorial the system provides an editor, that lets you write the tutorial text (black), record youir voice along with the lines, schedule visualizations (blue), set jumppoints (green) and control the tutorials progression, interaction, etc. via events (red).

The system generates an XML file that stores all the information about the tutorial, text. visualizations and events. It also creates a folder with soundsamples for the tutorial. How to use the LemmaAuthoringEnvironment.


In Lemma the representations on the visualization screen can be completely remodeled. There is a set of standard interactables which have a fixed location but the user is free to change the appearance of those items in a 3D modeler like 3ds max. More important is of course that the author can model any scene (with or without rigid bodys for rotational motion) and plug it into the application. In order for LEMMA to handle those geometries as intended there are a couple of simple guidelines to follow. They are explained in LemmaVisualizationModeling.

Known Issues & Pending Bugs

  • Every few hours the program crashes for no particular reason. The bug is located in the native code what you can tell from the non-Java error message. The villan is javaosgText.dll in 90% of the cases. This library seems to be pretty buggy. As LEMMA is storing the history of the current user every minute, it is sufficient to restart the program with "Last known position" set in the Configuration at the start.
  • Sometimes (?!) The vectors point in the opposite direction. Pressing "Reset Scene" usually solves the problem.


The following issues are problems that are very open for discussion as different approaches are possible. So if you have an idea please leave a comment. Don't be afraid to contribute, you can't "destroy" anything and I would appreciate it a lot.

  • Problem: How to provide a linear forward/backward in a document that does not only contain text, but also time depending events (e.g wait 4 seconds, then jump somewhere). This is critical especially when the script branches according to user input.
    • Current Approach: Stops the progression of the tutorial and just let the user navigate between the text lines. When the user reaches the point where she wants to proceed, she signals that to the application (by button, speech...). Then the program "jumps" to the new location and loads all events anew. (See LemmaAuthoringEnvironment#Tips)
    • Approach: A variant would be to do that only for the backward case as "forward" might be less critical (all events are processed in the normal progression).
  • Problem: Furthermore as you can jump anywhere, the element the user jumps to can also be an event that issues a wait event. In this case it must be assured, that the user CAN unblock the wait event and knows HOW to do this, without the context.
    • Approach: Assuring that the current text is displayed on the WhiteBoard and 'hoping' that everything works out.
    • Approach: Automatically go to the next line if the target is an event.
    • Approach: Automatically go to the previous line if the target is an event.
    • Approach: Implement sth. like a compact if_then_else construct thatsets a flag for the specific line and continues if the specific task has been performed before. This would be the nicest possibility, but
      • must be carefully designed to be 1) simple and 2) easy to use
      • if the user didn't get the point the first time he/she will not have that break at the second time
    • Current Approach: Avoiding to block the progression of the lecture due to various reasons. Despite exceptions where it is obvious what to do (e.g. Multiple Choice) always use the "lectureStopped" switch to pause the lecture.
  • Problem: how to provide a generic "Formula - GUI". It would not be a problem to hardcode such a gui, but making it applicable for ANY formula is a problem that afford mature programs like LaTeX or MathLab. The second problem is that actually realtime numbers flowing through formulas might be too much for the user. Lots of things moving simultaneously is very confusing.
    • Approach: We won┤t use a "Formula" GUI that displays calculations in realtime, but rather a pictureviewer. The author then just designs a picture of a formula (from a screenshot or drawing) and can display it on the Whiteboard.


Features in upcoming versions

  • speed up / slow down the lecturing or skip / rewind
  • make spoken and written / drawn annotations
  • adapt to the learners specific needs (e.g. omit voice if undesired)
  • logging the students progress
  • secondary content area
  • phycons like a spinning wheel could be augmented with e.g. forces
  • ...

Challenges outstanding from the topic

  • The JavaOSG by NoodleGlue is nice, but has a major drawback. One of the most important calls, getMatrix() in MatrixTransform produces a memory leak in allocating memory for a matrix but not freeing it again. Here you can find out more about JavaOpenSceneGraph, how to set it up use it and also a workaround for the mentioned problem.


A working version of LEMMA (for Windows) can be downloaded here. (Version 1.0 -- MarcBreisinger - 14 Feb 2006) Java 1.5 Runtime Environment must be installed. Besides that there should be nothing more to do than doubleclick start.bat. (If you already have OpenSceneGraph installed you may have to remove javaosgDB.dll from the working directory if you encounter problems at the start) The LemmaManual gives a short introduction on how to use the system. For using the system you need a 3D input device such as a tracker, a 3D mouse or sth. similar. The cheapest option is a P5DataGlove. You can download a server application for it here. The current solution is not optimal regarding performance and a improved version will be coming up soon. Till then it's a good idea to have an own computer only for the glove. The server is sending the data of the glove to port 3333 (or whatever port is specified in P5_Server.bat) and LEMMA can be configured to connect to this computer and port (Configuration -> port & host). Possibilities for other input devices will follow.


General Overview:

Quicktime MOV, 61 MB (slightly better quality)

DivX AVI, 16 MB

Interaction Techniques:

DivX AVI, 11 MB


FourEyesLab LEMMA Project Page including the results of the first user study with LEMMA

FourEyesLab UCSB

FogScreen Project Page


If you have suggestions, criticism, ideas about the project, please share them here:

Topic revision: r75 - 16 May 2007, RainerFink
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Medieninformatik-Wiki? Send feedback