Wednesday, October 8, 2014

smooth mobile scrolling

All GUIs generally work the same way.  There is a main thread with a loop that processes messages from a queue.  Messages can range from "move view to this location" or "user has performed a touch at location".  The whole point is that it is a queue so every message generally gets processed one at a time and in a first come first serve order.

For the majority of UI toolkits, including those found on iOS and Android, accessing and modifying objects must be done in the main thread.  Despite sometimes being called the UI thread, it is usually also the main thread and often is responsible for not just painting, changing colors, moving objects but also for loading files, decoding images, handling network responses etc.

In Android, if you want to animate an object and make it move an object from location1 to location2, the animation API figures out the intermediate locations (twinning) and then queues onto the main thread the appropriate move operations at the appropriate times using a timer.  This works fine except that the main thread is usually used for many other things -- painting, opening files, responding to user inputs etc.  A queued timer can often be delayed. Well written programs will always try to do as many operations as possible in background (non main) threads however you can't always avoid using the main thread.  Operations that require you to operate on a UI object always have to be done on the main thread.  Also, many APIs will funnel operations back to the main thread as a form of thread-safety. It is usually almost impossible to keep all operations on the main thread down to 1/60th of a second in order to allow animations to be processed smoothly.  Even if Google could manage to get their code to do just that, it doesn't mean third party Application writers will be able to.

In iOS operations on UI objects also must be done on the main thread with the exception of animation operations done via CoreAnimation.  CoreAnimation runs on a background thread and is able to directly manipulate, move, recolor and reshape UI objects on a background (CoreAnimation) thread.  Compositing, rendering is also performed in this thread.  It does this through a combination of hardware and software, providing very smooth and fast animations.  From the main thread you can basically issue a call to CallAnimation and tell it to move object1 from location1 to location2.  This animation will continue to run even if the main thread is blocked performing another operation.  This is why animations will almost never stutter on iOS.The main thread manages application data and UI application state (UI application state includes things such as the strings to be displayed in a ListView etc) but issues physical UI state change requests to a separate and dedicated high priority CoreAnimation thread (physical states include things such as color, position and shape). All physical state changes can be animated and CoreAnimation will also perform the twinning for you (like the Android animation APIs).  Non animated physical state changes will be issued directly by CoreAnimation  and the main thread (not the CoreAnimation thread) will block until those are performed.  Animated physical state changes that are issued by the main thread will be performed asynchronously by the CoreAnimation thread. Because physical UI state and only physical UI state is managed by the CoreAnimation thread, the main thread can be blocked or busy but the CoreAnimation thread will still continue to not only accurately render the last known state of the UI (as issued by the main thread) but also continue to render any pending or incomplete animated UI physical state changes as requested by the main thread.

In Windows Vista, Microsoft introduced desktop composition whereby the OS maintained a separate pixel buffer for every window.  This meant that even if an application hung, the last state of the window (how it looked) is still rendered rather than just being drawn as white (the OS partially managed the state of the pixels in the window).  CoreAnimation goes beyond this and offloads much of the UI work traditionally managed by the main thread including managing the state of not just the pixels (like Vista) but of higher level concepts such as widgets, widget locations, widget colors etc.

At Android animation model, It's the way many toolkits work including Flash which was definitely very animation heavy.  I would say the iOS model makes the overall user experience nicer and offloads one more worry for the developer back to the operating system.  I'm sure Google will continue to recognize the importance of animation on touch screen devices and continue to accelerate (or re architecture) Android in coming releases.

A 5 year old 1st generation iPhone will perform smoother and more reliable animations than the latest quad core Samsung Android phone. It's a software design problem and not something you can throw more cores at (not least of which because the main thread will only ever run on one core!). Don't believe people when they excuse stutter and lag as "oh just the Android Java garbage collector".  Modern compacting, generational garbage collectors generally aren't the cause of the kind of stutter you see on Android.


Let’s not forget that Android does true multitasking and background processes execution, unlike iOS, which adds to the overhead. Also adding to the overhead: Native iOS apps are binaries pre-compiled for their own hardware while Android uses the Dalvik virtual machine with just-in-time compilation to run Dalvik dex-code (Dalvik Executable), which is usually translated from Java bytecode.

Kinetic scrolling

Kinetic scrolling is the combination of regular, drag-finger-on-screen scrolling with an additional movement after the finger is lifted off the screen.Based on how fast the finger was dragged on the screen, the duration, speed and deceleration of the additional movement can vary.

kinetic scrolling can be viewed as the sum of two features, it can be implemented in two steps.

The first step is click & drag scrolling. It can be achieved by installing an event filter and intercepting mouse press, move and release events. When a press event is received the scrolling starts, when a move event is received the list is scrolled, and finally when a release event is received the scrolling stops. To avoid accidental clicks, all the events are blocked inside the filter function.

For step two, the scroller continues to scroll the list automatically after the user has lifted their finger off the screen, gradually slows down and then stops. To display a pleasing effect, the scroller must decide how fast to scroll, how far to scroll and how fast to slow down.

A good starting point is "how fast to scroll". In physics velocity represents the direction in which and magnitude by which an object changes its position. Speed is another word for magnitude in this context. The "how fast to scroll" question can be answered by recording the cursor’s drag velocity on the screen. A simple but imprecise way to do this is to poll the cursor position at specific time intervals; the difference in positions represents the speed (measured in pixels / timer interval) and the mathematical sign of the difference represents the direction. This algorithm will give a good enough idea on whether the cursors is moving fast or slow.

Next up is "how far to scroll".How far is actually connected to how fast to slow down because the list is scrolled with a certain velocity and then it decelerates until it stops. Since the velocity has previously been established, the only thing left is to calculate the deceleration based on friction. In physics, kinetic friction is the resistance encountered when one body is moved in contact with another. Of course, there can be no friction between pixels, but kinetic scrolling is a simulation and one can pretend that the list items are moving over the list container and that this movement generates friction. In reality friction is calculated based on the nature of the materials, mass, gravitational force and so on. In the simulation a numeric value is used to alter the speed of scrolling.

Having determined the deceleration,"how far" the list scrolls kinetically is simply a function of the time that it needs to reach a speed of zero.

Final Steps:
  • Need to measure the velocity of finger cursor.
  • Implement a simple particle physics loop. information about how to do that here
  • give your particle "bounds" using math derived from the width of your scrolling plane, and the width of your viewport
  • continuously Add the the difference between the mouse velocity and the particle velocity, to the particle's velocity, so the particle's velocity "matches" the mouse's velocity for as long as it's moving.
  • Stop doing step 4 as soon as the user lifts their finger. The physics loop takes care of inertia.
  • Add your personal flourishes such as "bumper" margins, and smooth scrolling "anchor" points
Basic scrolling functionality: This task consisted in handling drag events in the simplest way, which is by scrolling the contents according to the drag distance and direction. This was relatively straightforward to implement, the only tricky aspect was to know when to start a drag operation vs. when to pass down a tap event to a child widget inside the scrollable area.

Scroll inertia: This one was the most challenging. The idea here is that scrolling should continue for some time after the user lifts the finger, slowing down until it stops completely. For this I needed to have an idea of the scroll velocity. Unfortunately it is not accurate to compute the velocity from a single sample, so while the user is scrolling I record the last N motion events in a circular buffer, along with the time at which each event occurred. I found N=4 to work just fine on the iPhone and on the HP TouchPad. When the finger is lifted I can compute an approximate start velocity for the inertial scrolling from the recorded motion. I defined a negative acceleration coefficient and used standard motion formulas (see here) to let the scrolling die down nicely. If the scroll position reaches a border while still in motion I just reset the velocity to 0 to prevent it from going out of range (the abrupt stop is addressed next).

Flexible scrolling limits: instead of going into an abrupt stop when the scroll reaches the end I wanted the widget to scroll some, but offering resistance. For this I extended the allowed scroll range on both ends by an amount that I defined as a function of the widget dimensions. I've found that adding half the width or height on each end worked nicely. The trick to give the scrolling the feeling that it is offering some resistance was to adjust the displayed scroll positions when they are out of range. I used a scaling down plus a deceleration function for this (there are some good easing functions here).

Spring behavior: since now it is possible to scroll past the valid range, I needed a way to bring the scroller back to a valid position if the user left it out of range. This is achieved by adjusting the scroll offset when the scroller comes to a stop at an out of range position. The adjustment function that I've found to give a nice springy look was to divide the distance from the current position to the desired position by a constant and moving the offset by that amount. The bigger the constant the slower motion.

Scrollbars: the final touch was to add overlay scrollbars, which fade in when scrolling starts and fade out when it ends.

Quick list for android:
  • Reduce the number of conditions used in the getView of your adapter.
  • Check and reduce the number of garbage collection warnings that you get in the logs
  • If you're loading images while scrolling, get rid of them
  • Set scrollingCache and animateCache to false (more on this later)
  • Simplify the hierarchy of the list view row layout
  • Use the view holder pattern
  • Use dragging proper way (is the type of scrolling that occurs when a user drags her finger across the touch screen Simple dragging is often implemented by overriding onScroll() in GestureDetector.OnGestureListener.)
  • Use flinging proper way (is the type of scrolling that occurs when a user drags and lifts her finger quickly. After the user lifts her finger, you generally want to keep scrolling (moving the viewport), but decelerate until the viewport stops moving. Flinging can be implemented by overriding onFling() in GestureDetector.OnGestureListener, and by using a scroller object.)

No comments:

Post a Comment