name: inverse layout: true class: center, middle, inverse --- # Event Handling IV - View Updates and Essential Geometry Lauren Bricker CSE 340 Winter 2023 --- layout:false [//]: # (Outline Slide) # Today's goals - Input dispatching process (toolkit architecture) - Picking (alternative: Focus) - Capture - Bubble - Propositional Production Systems in implementing interactors - Essential Geometry - Menus Assignment Administrivia - Menus out (one week of coding, 1 week of doing user study) --- # Midquarter Eval feedback Thank you for participating in the evaluation! Things we "heard" - OH, Sections, Discussion board, Lecture recordings and activities are helpful. - Multiple select practice quiz answers frustrating - Limited by Ed, but we'll find another way to present those questions. - Doodle and Layout were very time consuming - We'll look at **scaffolding** but not just give the answers - Note, every quarter we update the specs with info and hints based on prior offerings. - Pre-class readings - We limit the number of required ones, optional ones are resources for the assignments. --- # Midquarter Eval feedback **Reminders** - This is NOT an Android programming class. It is an interactions programming class using Android. - Reflections are NOT an afterthought. - Design work must be done before the creative part. - Your work on reflecting what you could and couldn't get done will help you with your final project. I was a software engineer and manager for YEARS - I know this works. --- template: inverse ## Review: Input Dispatch Process --- # Thinking at the Toolkit Level .left-column[ <div class="mermaid"> graph LR ap[Application Program] hlt[High Level Tools] t[Toolkit] w[Window System] o[OS] h[Hardware] class ap,o,h,hlt yellow class w,t green </div> ] .right-column[ Windows System: Tells which window received the input Toolkit Architecture: - Receives events from windowing system - Packages up Events and determines which `Views` should get them - Calls Callbacks - Handles other aspects of MVC (like redraw) ] --- layout: false # Review: Input __Dispatch__ Process Input thread: - When a user interacts, __events__ are created - Events go into a queue ??? What do you think of when you hear the word "thread"? How does it relate to CS? Remember a queue is a data structure that is first in first out. --- ## How does the Toolkit Architecture deliver events? Application’s UI Thread (the main thread of your program) ```java while (true) { // Gets event from the application message queue. // If the queue is empty, we just wait here. Event e = GetEventFromMessageQueue(); // Dispatches the event to the appropriate application window // (the destination address is in the event data). DispatchEventToMessageDestination(e); } ``` This code is automatically setup and managed by the UI framework you are using --- # Review: Input __Dispatch__ Process Input thread Dispatch thread: - Front event comes off queue - Toolkit decides where to send events based on Focus or Positional input? - Focus list (in order based on interest) - Positional list (z-order under cursor based on interactor hierarchy) - Event gets sent to a listener (callback) registered with a `View` or `Activity`. - The `View` or `Activity` has to implement the callback (in the Application) - otherwise the toolkit handles this in the "default" way --- ## Event Dispatch (theory) .left-column-half[ ] .right-column-half[ Dispatch Strategies (theory) - Positional - Bottom-up - Top-down - Bubble out - Focus-based ] --- ## Event Dispatch (theory) .left-column-half[ ] .right-column-half[ Dispatch Strategies (theory) - Positional - Bottom-up - **ok** - Top-down - **ok** - Bubble out - Focus-based - **ok** Please review the previous lecture if not **ok** ] --- ## Event Dispatch (theory) .left-column-half[  ] .right-column-half[ - Bubble-out - Used when there is no clear nesting of interactive objects - Tree is traversed as in top down approach, but bounding rectangles are hints, not guarantees - Objects checked in opposite order from drawing: frontmost items are checked first - Object that was hit is attached to event, ancestors know what was selected ] --- # Events Dispatch (in Android) .left-column50[  ] .right-column50[ * `MotionEvent` (which inherits from `InputEvent` is passed through these methods) * At a raw level, `View` gets an `onTouchEvent` * `MotionEvent#getAction` will return actions such as `ACTION_DOWN`, `ACTION_MOVE` * `onTouchEvent` can be handled directly or the system may combine these to be an `onClick` if you've registered a `View.OnClickListener` (for example) ] .footnote[From [Android Touch System — Part 1: Touch Functions and the View Hierarchy](https://proandroiddev.com/android-touch-system-part-1-touch-functions-and-the-view-hierarchy-1f6526e55d78)] --- # onTouchEvent We described how to set up `View.OnClickListeners`... Now we're focusing on handling the `View#onTouchEvent` callback... -- count: false `onTouchEvent` will allow us more fine grain control in event processing. -- count: false We'll be able to create "odd shaped" interactors. -- count: false Understanding this will be the key to our next assignment. --- # Dispatch Process (Positional) .column[ <div class="mermaid"> graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6)) class V0,V1,V2,V5,V4,V3,V6 blue </div> ] .column[  ] --- # App Developer's Scenario .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> ] .column[  ] .column[ - `V6` Not touched because it is outside the "interactable area" in the bounding box - `V4` Doesn't react because it is a decoration - `V3` Doesn't react because it is a decoration - `V1` Reacts and moves because it is a draggable container ] ??? Circular interactor! Like Pie Menu --- # Step 1: Picking .column[ <div class="mermaid"> graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6)) class V0,V1,V2,V5,V4,V3,V6 blue </div> ] .column[  ] .column[ Android traverses the view hierarchy (starting at root) "Picks" all `View` objects that respond to an input event and puts them into a list ] --- # Step 1: Picking .column[ <div class="mermaid"> graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6)) class V0,V1,V2,V5,V4,V3,V6 blue </div> {`V6`, `V5`, `V4`, `V3`, `V2`, `V1`, `V0`} ] .column[  ] .column[ Picking is a traversal of the tree - Order matters! - It uses reverse draw (z) order ] --- # Step 1: Picking .column[ <div class="mermaid"> graph TD V0((V0)) --> V1((V1)) V0 --> V4((V4)) V1 --> V2((V2)) V1 --> V3((V3)) V4 --> V5((V5)) V4 --> V6((V6)) class V0,V1,V2,V5,V4,V3,V6 blue </div> {`V6`, `V5`, `V4`, `V3`, `V2`, `V1`, `V0`} ] .column[  ] .column[ But the system refine and filter this! What are some reasons to skip views? ] ??? - Not inside the view - View doesn't care about that type of input --- # Step 1: Picking (and Filtering) .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> {`V6`, `V4`, `V3`, `V1`, `V0`} ] .column[  ] .column[ - Toolkit *may* discard components that do not contain the event's position - Some toolkits check with components for example by calling `pick()` on each component (not Android) ] .footnote[*: denotes the element responds to the event because of position/type] --- # Step 2: Delivering .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> {`V6`, `V4`, `V3`, `V1`, `V0`} ] .doublecolumn[ **Top-down**: Walk the pick list right to left (reverse z order). Sometimes called *Capture* **Bottom-up**: Walk the pick list left to right (approximately corresponds to z order). Sometimes called *Bubble* Some toolkits do both (with different listeners per event type), some only bottom-up. Most events can only be *consumed* by a single component (event delivery returns a `boolean` to say event is used or not, e.g `onTouchEvent()`)] --- # Step 2: Delivering .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> {`V6`, `V4`, `V3`, `V1`, `V0`} ] .doublecolumn[ Android mostly uses Bottom Up (bubble) - Dispatch starts at the start of the picked `View` object list (`V6`) - Walking down the list, dispatch asks: will you consume this event? (e.g. by calling `onTouchEvent()`) - If `true`: the event is consumed and the event propagation __stops__ - If `false`: Move to the next element in the `View` list ] --- # Step 2: Delivering .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> {`V6`, `V4`, `V3`, `V1`, `V0`} ] .column[  ] .column[ Scenario: - `V6` returns `false` because outside the true interactor area - `V4` and `V3` return `false` because they are a decoration - `V1` returns `true` because it is a draggable container **Dispatch halts and redraw starts** ] ??? Circular interactor! Like Pie Menu --- # Positional Input - Recap .column[ <div class="mermaid"> graph TD V0((V0*)) --> V1((V1*)) V0 --> V4((V4*)) V1 --> V2((V2)) V1 --> V3((V3*)) V4 --> V5((V5)) V4 --> V6((V6*)) class V2,V5 blue class V0,V1,V4,V3,V6 bluegreen </div> {`V6`, `V4`, `V3`, `V1`, `V0`} ] .column[  ] .column[ |*Picking* | V6| V4 | V3 | V1|V0| |:---------------|:-------------------| |*Capture Order* | 5 | 4 | 3 | 2 | 1| |*Bubbling Order*| 1 | 2 | 3 | 4 | 5| - Android uses bubbling - Picking and Delivery are based on interactor hierarchy - A view can refuse an event (return `false`) - Dispatch halts when event is consumed ] --- # Stuff to ponder How would you handle focus input? - Some event types are focus based (e.g. keyboard input) - Just skip picking: Focus list is globally created, and we walk it the same way -- Implementing Drag and Drop - Focus? Or Positional? -- - [Android's tutorial is positional](https://developer.android.com/training/gestures/scale#java) -- Scrollbars - Focus? Or Positional? -- - both --- # How does *Android* decide where to send events? - **Capture** Android does have *some* support for this - `onInterceptTouchEvent()` in Android - **Pick** to identify objects of interest (or just focus()) - `buildTouchDispatchChildList()` in Android; happens only after capture! - not generally accessible but you can override it in a custom interactor - **Bubble** Standard in Android - example: `onTouchEvent()` in Android --- # What about callbacks? Event delivery in view happens using callbacks - e.g. `onTouchEvent()` is a callback (from the `View.OnTouchListener` interface) -- For applications, - Usually Views invoke callback (when interaction is complete) - This is part of your *controller* code in MVC parlance - We often implement this by creating a custom listener in Android - Can also create a custom event but not required (we'll do this in `Undo`) .footnote[fascinating to look at [implementation in ViewGroup](https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/view/ViewGroup.java) -- not very general!] <!-- --- --> <!-- # Note about event records --> <!-- - Where --> <!-- - Where is the location of the cursor (x,y position on the screen) --> <!-- - For focus based events, where is often the target --> <!-- - Value is anything you might need to know (like whether it's a 1 or 2 finger swipe; or which key was pressed) --> <!-- Why is this important? For *picking*: --> <!-- - When input is positional, *only* interactors whose bounding box includes (x,y) -- where -- are picked --> <!-- - When input is focus based, *only* interactors in the focus list are picked. Where isn't used in dispatch --> --- name: inverse layout: true class: center, middle, inverse --- # Using Input to Create Interaction Techniques ## State Machines, PPS, Essential Geometry --- layout:false # Interaction Technique A method for carrying out a specific interactive task For example: For entering a number in a range you could use... ??? have the class try to think of examples -- - (simulated) slider - (simulated) knob - type in a number (text edit box) Each is a different interaction technique --- # Example: Specify the end points for a line Could just specify two endpoints – click, click - not good: no affordance^* - no feedback (/ feedforward) Better feedback is to use “rubber banding” - stretch out the line as you drag - at all times, shows where you would end up <br> if you “let go” .footnote[^*We will learn about this in week 7] ??? Importance of feedback vs application callback --- # Implementing rubber banding ``` Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ??? Discuss! Not event based Not in the basic event/redraw loop Potentially locks up the system --- # Implementing rubber banding Need to get around this loop <br> absolute min of 5 times / sec – 10 times better – more would be better but might block system. ??? aside -- why 5-10 times per second? --- # Event driven code Needs to respond to events as they arrive Needs to maintain state between events --- # Solution: State Machine .left-column50[ - State Machines are used to describe responses to incoming events and can store state between events. - **Start state** - incoming arrow - **End state** - double-layered shape (means "reset to start") - **Transition States** - single-layered shape - **Event Arrows** - between states, represent different actions taken up states. ] .right-column20[ <div class="mermaid"> stateDiagram-v2 [*] --> A: a A --> B: a B --> B: b B --> C: b C --> [*]: a </div> ] -- .right-column30[ Which of these strings are "accepted" by the state machine? (Answer on [Ed](https://edstem.org/us/courses/21053/lessons/31178/slides/191151)) 1. ab 2. abbbbbb 3. aaba 4. abbbbba 5. aabbbbbbba 6. aaaabbbaa ] --- # Solution: State Machine State Machines are used to respond to incoming events and allows for us to store state between events. .left-column60[ - Actions always take place on arrows (*transitions*) - Start state - indicated in black - End state - indicated with double-layered shape (means "reset to start") - Transition States - indicated with single-layered shape - Event Arrows - indicated with an arrow between states, represent different actions taken up states. ] .right-column40[ <div class="mermaid"> stateDiagram-v2 [*] --> B: Event1/?Callback1() B --> B: Event2/?Callback2() B --> [*]: Event3/?Callback3() </div> ] --- # Propositional Production System (PPS) - PPS is a state machine with extra conditions required to fire on the arrows. - ? predicate (Boolean expr) before event, adds extra conditions required to transition - action calls - Typical notation `event / pred ? action()` - Example `MouseUp/inView?Start_Line()` - Finite State Machine (FSM) augmented with guards is Turing complete --- # PPS Example: Rubber Banding .left-column60[ Compare to previous implementation: ```txt Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ] .right-column40[ - Determine the Events (triggers) - Determine the States - Determine the Actions - Determine the Queries ] --- # PPS Example: Rubber Banding .left-column60[ Compare to previous implementation: ```txt Accept the press for endpoint P1; P2 = P1; Draw line P1-P2; Repeat Erase line P1-P2; P2 = current_position(); Draw line P1-P2; Until release event; Act on line input; ``` ] .right-column40[ <div class="mermaid"> stateDiagram-v2 [*] --> Drawing : Mouse Down/inView?Start_Line() Drawing --> Drawing : Mouse_Move/inView?Update() Drawing --> [*] : Mouse_Release/inView?Finish_Line() </div> ] --- # PPS Example: Rubber Banding .left-column50[ Reading a state machine: translates input sequence into action! - When you are in Start State, and a `Mouse_Down` event arrives, do the action `Start_line()` and go to Drawing State. - Update the line end point position every time the mouse moves. - When it releases (`Mouse_Release` event), finish the line (at this stage a callback to the application might be appropriate) ] .right-column50[ <div class="mermaid"> stateDiagram-v2 [*] --> Drawing : Mouse_Down/inView?Start_Line() Drawing --> Drawing : Mouse_Move/inView?Update() Drawing --> [*] : Mouse_Release/inView?Finish_Line() </div> ] ??? How could we provide a better affordance? Does it matter if we are using a mouse or a touch screen? --- name: inverse layout: true class: center, middle, inverse --- # Essential Geometry as the basis for state --- layout:false # Essential Geometry as the basis for state .left-column-half[  ] .right-column-half[ - What is the essence (or nature) of this scrollbar? Where can you interact with it? ] --- # Essential Geometry as the basis for state .left-column-half[  ] .right-column-half[ - What is the essence (or nature) of this scrollbar? Where can you interact with it? - on the thumb - inside the scrollbar above the thumb - inside the scrollbar below the thumb - How do we implement a scroll bar like this? ] --- # Step 1: Essential Geometry Implementation - An `essentialGeometry(MotionEvent event)` method converts position to geometry - `essentialGeometry` methods are good practice but not required by the Android Toolkit - `essentialGeometry` returns an **enum** that tells you what part of the geometry you are in for that point (such as `Inside` and `Outside`) --- # Aside: Enums Enums are a group of named constants - Often used for developing interactions for defining PPS States and Essential Geometry - Will be used in the Menu assignment for interaction *and* for experimental conditions - Good code quality: limits choices of what a variable can be. [Documentation](https://docs.oracle.com/javase/tutorial/java/javaOO/enum.html) --- # Step 2: Design the PPS with essential Geometry .left-column30[ <div class="mermaid"> stateDiagram-v2 [*] -->Scrolling: A [*] --> [*]: B [*] --> [*]: C Scrolling --> Scrolling: D Scrolling --> [*]: E </div> ] .right-column70[ ```java // A MouseDown / inThumb ? startScroll() // B MouseClick / InsideAboveThumb ? scrollUp() // C MouseClick / InsideBelowThumb ? scrollDown() // D MouseMove / updateThumbDocument() // E MouseUp / keepLocation() ``` ] --- # How do we implement this? - Implement in `onTouchEvent()` using a switch statement - Assume there is an `essentialGeometry(MotionEvent event)` method. - This returns a enum that tells you what part of the geometry you are in for that point. - Assume implementations of all the methods given (i.e. `startScroll()`, `updateThumbDocument()`...) - Assume a field, `mState` which is the current state of the state machine - We are comparing against enums for `EssentialGeometry` and `State` --- # Step 3: Translate the PPS to code ```java public boolean onTouchEvent(MotionEvent event) { EssentialGeometry geometry = essentialGeometry(event); switch(mState) { case START: ... // return true if handled break; case SCROLLING: ... // return true if handled break; default: break; } return false; // not handled! } ``` --- # Step 3: Translate the PPS to code - START ```java case START: if (event.getAction() == MotionEvent.ACTION_DOWN && geometry == EssentialGeometry.INSIDE) { mState = State.SCROLLING; updateThumbPosition(); updateThumbAlpha(); updateVolume(); invalidate(); return true; } break; ``` --- # Step 3: Translate the PPS to code - SCROLLING ```java case SCROLLING: if (event.getAction() == MotionEvent.ACTION_MOVE && geometry == EssentialGeometry.INSIDE) { updateThumbPosition(); updateVolume(); invalidate(); return true; } else if (event.getAction() == MotionEvent.ACTION_UP) { mState = State.START; updateThumbAlpha(); invokeVolumeChangeListener(); invalidate(); return true; } // otherwise fall through and return false break; ``` --- # Let's try it for a button How to get started - Determine the Events (triggers) - Determine the States - Determine the Queries (essential geometry, context) - Determine the Actions --- # Let's try it for a button .left-column70[ - Essential geometry - Inside (button) - Outside (button) - States are: - PRESSED - NOT_PRESSED - Methods that exist - `indentButton()` (when button is pressed) - `normalButton()` (when button is not pressed) - `invokeAction()` (when the user releases in the button) - `cancelAction()` (when the user releases outside the button) ] --- # PPS for a button .left-column50[ - Essential geometry - Inside (button) - Outside (button) - States are: - PRESSED - NOT_PRESSED - Methods that exist - `indentButton()` - `normalButton()` - `invokeAction()` - `cancelAction()` ] .right-column50[ <div class="mermaid"> stateDiagram-v2 [*] --> PRESSED: DOWN / indentButton() PRESSED --> NOT_PRESSED: MOVE / <BR> Outside ? normalButton() NOT_PRESSED --> [*]: UP / cancelAction() PRESSED --> [*]: UP / invokeAction() NOT_PRESSED --> PRESSED: MOVE / <BR> Inside ? indentButton() </div> ] --- # Code for the button ```java public boolean onTouchEvent(MotionEvent e) { EssentialGeometry geometry = essentialGeometry(event); switch (mState) { case State.START: // pps arrow from start to INSIDE break; case State.INSIDE: // need both arrows from pps case State.OUTSIDE: // need both arrows from pps default: break; // ignore everything else } return false; } ``` --- # Code for the button ```java @Override public boolean onTouch(MotionEvent e) { EssentialGeometry geometry = essentialGeometry(event); switch (mState) { case State.START: if (e.getAction() == MotionEvent.ACTION_DOWN && geometry == Geometry.INSIDE) { indentButton(); mState = State.PRESSED; return true; } break; case State.PRESSED: ... ``` --- # Code for the button ```java case State.PRESSED: case State.NOT_PRESSED: ... ``` --- # How do we implement this? ```java case State.PRESSED: if (e.getAction() == MotionEvent.ACTION_UP) { invokeAction(); mState = State.START; return true; } else if (e.getAction() == MotionEvent.ACTION_MOVE && geometry == Geometry.OUTSIDE) { normalButton(); state = State.OUTSIDE return true; } break; case State.NOT_PRESSED: if (e.getAction() == MotionEvent.ACTION_UP) { cancelAction(); mState = State.START; return true; } else if (e.getAction() == MotionEvent.ACTION_MOVE && geometry == Geometry.INSIDE) { indentButton() state = State.INSIDE return true; } break; default: break; ``` --- # When to use PPSs You're probably already using them, just not intentionally (and maybe less well as a result) PPSs are a good way to do control flow in event driven systems Can do (formal or informal) analysis - are all possible inputs (e.g. errors) handled from each state - what are next legal inputs: can use to enable / disable Can be automated based on higher level specification --- # Summary State machines are very good (for this job) but do have limits State machines don't handle independent actions very well (state explosion) Mostly useful for smaller things - Great for individual components - Not so great for whole dialogs Path of least resistance is rigid sequencing Ask: is this good for what I am doing? ???