Even a chimp can write code

Monday, April 14, 2008

Silverlight 2: Layout and Rendering engines

A layout engine which does precise and optimal size or position calculations is critical in any user interface framework for building responsive, interactive and high-performance applications. The term "layout engine" is often interchangeably used with "rendering engine" to mean code that takes markup content along with styles, templates and other formatting instructions to display content on screen. As prominent examples, you know of Trident, the layout engine in Internet Explorer, or Gecko which performs a similar function in Netscape/Mozilla family browsers.

In Silverlight, we typically look at the layout and rendering engines as two distinct things rather than one amorphous entity. The former handles the math which dictates how UI elements are sized and positioned on screen, and how they may be constrained or clipped in order to fit into the bounding box. The latter deals with the logic of taking bits representing graphical objects and showing them on screen. Lets do a quick lap around these features.

The Silverlight layout engine

Among the goals of the layout engine is to:

  • Enable highly complex and dynamic user interfaces where elements can be automatically positioned
  • Provide a gradient of Panel choices in which to place UI elements, as well a mix of knobs to achieve very precise positioning: absolute/relative sizes, margins, padding, static or specified sizes as well as dynamic or calculated sizes
  • Allow app code to participate in layout. This can be as simple as being notified on layout events, or as niche as extending the layout behavior by actively participating in the process

With the design patterns and public APIs of the layout engine in Silverlight, we've stayed close to the behavior displayed by Windows Presentation Foundation (WPF), even if the implementation is custom keeping in mind the needs and constraints of Silverlight.

Very simply put, layout in WPF or Silverlight is a recursive operation that involves two passes, measurement and arrangement. During measurement, a child element is effectively told by its parent element "Here's how much room is available to all you kids and your 'gear'. Tell me how much you really need." Once all appropriate elements in the tree have been measured in this way, the arrangement pass kicks in. At this time, the elements' sizes are finalized. The parent element can choose to say either "Sure you can have your own room" or if the demands are profligate, then "Oh, the kids these days! I'd like champagne on a beer budget too. But no can do. Get your things together - you and your brother are rooming up." After arrangement, the elements in the tree are ready to be rendered.

Example of a grid with constrained size. The rectangle within it asks the parent for all its got -- and gets it: 
<Grid Width="250" Height="600">
      <Rectangle Fill="Red" />

Example of a stack panel with constrained size. The first rectangle gets what it desires. The second rectangle gets clipped because it desires more than the parent has room for:

<StackPanel Width="250" Height="600">
      <Rectangle Fill="Red" Width="250" Height="300"/>
      <Rectangle Fill="Green" Width="300" Height="300" />

Needless to say, the layout algorithm employs optimizations to ensure layout doesn't happen until absolutely necessary, and when it does, there selectiveness on which items need to be considered. I won't go into too much detail but primarily this keys off the knowledge that certain properties such as Width, Height, Visibility, Stretch, Alignments, Margins etc. are deemed layout-affecting and a change to one or more of these invalidates the current layout, causing Silverlight to re-layout i.e. measure and arrange its UI elements.


The Silverlight rendering engine

Silverlight uses vector graphics as its rendering data format, giving its graphics the ability to scale to any size or display resolution.

In WPF, the Visual class is the basic abstraction that provides rendering support. UI controls like Button, ListBox etc. derive from UIElement and FrameworkElement, which derive from Visual. In Silverlight we've dispensed of the Visual type, folding its functionality into its child types. This way we can keep the object hierarchy shorter, have one less type to deal with, small size reductions and still be a subset of WPF both conceptually and behaviorally. Silverlight also generally discards the notion of distinct visual and logical trees, effectively reducing to one tree where visual/drawing/logical info is persisted. This tree is the effective consolidation of elements you created in markup or code, containing all rendering info needed to output to screen. The tree determines rendering order as well - starting with the root visual (the top-most node), progressing to its children from left to right. If an element has children, then they are traversed before its siblings.

Silverlight uses retained mode rendering and employs a custom rendering pipeline, not relying on OS libraries like GDI/GDI+/WPF for better consistency and performance across platforms. It uses occlusion culling among other approaches to prevent overdraw. This is particularly significant if you consider that a typical application has 3-4x overdraw and perhaps 10x if effects are employed. Another very significant (but not commonly known) fact about Silverlight's rendering engine is its many-core support; specifically how it was architected to utilize the raw power of multiple cores and subdivide rendering tasks to be processed in parallel by CPU cores as they have available time. This provides phenomenal linear speed increases when you have dual, quad or 8 core machines. In this regard, Silverlight has few peers. Of course, one must not gloss over the detail that other functions such as tree walks, layout, template expansions etc. do not necessarily scale linearly.


Blog posts are never a substitute for good documentation (and we're fortunate to have a great team of doc writers). My intent with this post was to mix the "how" with the "why". One of my mental models is to view teams based on what I call "team DNA". In this respect, the UI platform team in Silverlight traces its DNA to the User32, GDI/GDI+, Trident, WinForms and WPF teams (Note: I'm way oversimplifying - we have a far more diverse team in ideas, outlooks and heritage than I can describe with words; but bear with me here). I think the team's DNA gives it a rich corpus of experience in blending the web and desktop, plus the ability to distil things on the "what works", "what doesn't work" and "what didn't work then, but will now" buckets. In the spirit of evolution and due to the unique size and platform requirements, the layout and rendering engines as other areas in Silverlight, introduce optimizations and use different algorithms than their WPF cousins. Feedback is always appreciated.

Further reading:

Labels: , , , ,

Email this | Bookmark this


  • Thanks for the post.

    I really wish you guys posted the bits with the layout manager earlier. I get the feeling that it's really too late to feedback and change these differences now.
    I understand the download constraints but these differences in the core rendering/layout/databinding models are going to cause real pain in the long term future. The problem is, over time, these size constraints will be lifted and we'll end up with two different models that will always be different under the covers.

    Does taking a base class out really save space? I would have thought that it would actually increase the download size (duplicate functions in child classes).

    I still haven't used SL2 enough to fully realize the implications of a unified logical/visual tree. How do you walk the logical tree after data templates have been applied on a typical ContentControl? Does it just disappear and get replaced by the visual tree?

    Does this also firmly close the door on any Silverlight/WPF/MILCORE integration in the future? I was hoping to see some hardware accelerated features if the runtime discovers the right support on the machine. I can see the competition doing this - and I can see it being easier to do on the controlled environment of the Mac.

    I suppose I would have prefered for you to move out the controls or default temlates into some optional additional download and concentrate on parity with WPF for the plumbing. I really don't understand the unique requirements of the web being different to the desktop - other than download size. At the end of the day - it's application code just being deployed in a different way.

    By Blogger Joe, at April 14, 2008 at 5:37 PM  

  • I agree 100% with what Joe is saying. I wish you would spend more time on synchronizing WPF and Silverlight. The differences you are introducing are going to create a platform nightmare. Additionally, the multi-core capabilities of the software rendering engine are going to be meaningless when a future version of Flash arrives with true hardware acceleration. It would have been better spent trying to figure out how to do hardware acceleration from the beginning. I get frustrated whenever I think about this stuff, primarily because MS has done nothing to indicate that it is truly taking these concerns seriously; no matter how many developers keep raising the same issues.

    By Anonymous Rob, at April 14, 2008 at 6:26 PM  

  • Totally agree too! How can you get rid of the concept of the separate visual and logical trees? That is such a core concept in WPF! How will dynamic templating possibly work?

    Every time I read a blog entry on Silverlight, I get more frustrated. I understand that time constraints mean that all the features of WPF cannot currently be implemented in Silverlight... but most blog entries from the Silverlight team seem to indicate that they never will be!

    Despite being impressive at first, I really don't care about DeepZoom and things like that - they are only useful to a tiny subset of applications. Surely the eventual aim MUST to have all the same core concepts in WPF and Silverlight - this is afterall the plugin originally named WPF Everywhere!

    By Blogger Neil Mosafi, at April 15, 2008 at 7:46 AM  

  • The operative words are "not now", not "never". Silverlight will only be successful if it rises up to its stated vision and fulfils the desires of rich internet app authors.
    If you have specific complaints about incompatibilities, please let us know. We care about this deeply. Arguing in abstract terms is not useful. Let's take layout as an example. For the same markup or code, if Silverlight and WPF results are 100% similar on render, then do you care if their internal algorithms are different? If so, why?

    By Blogger Ashish Shetty, at April 15, 2008 at 3:29 PM  

  • Maybe I am wrong about the "never" thing...

    Anyway I could be more specific and list things like: No DependencyProperty metadata (e.g default values, inheritance etc), no {x:Type} or {x:Static} markup extension, no custom markup extensions at all, no resources at the FrameworkElement level, etc... but I think most of these points have been raised on previous blogs and forums.

    Regarding the lack of distinct logical and visual trees, does this not mean that once a DataTemplate/Style is applied (e.g. creating a visual tree for a logical element) then that is it and a new template cannot be reapplied at a later point?

    By Blogger Neil Mosafi, at April 16, 2008 at 3:46 PM  

  • I completely agree with Joe. I'm porting a WPF app to Silverlight now, and it's a HUGE pain (don't even get me started on LayoutTransform).

    By Blogger Mark, at July 7, 2008 at 2:28 PM  

  • There are two schools of thought that need to be reconciled here. One is Nicholas Carr's The End of Corporate Computing article in MIT Sloan Management Review. The other is Microsoft's Gianpaolo Carraro. Carraro believes in Software as a Service and believes the desktop model of licensing software is the way to go.

    The truth is that neither of these viewpoints address the technological issues we as developers deal with.

    Currently, my co-worker built in ONE DAY an application in Ext.js that has taken me two months to even come close to completing in SL2. If I didn't have to deal with missing features and incompatibilities with WPF, then I might have been done by now.

    It's pretty weird that you are accusing Joe of arguing in abstract terms. He asked a very straightforward question: How much space are you saving by taking out the Visual class?

    As developers, we need to hear from "The Eric Lippert of Silverlight/WPF" what is going on. I am comfortable about the direction of C# b/c Eric is always talking about how careful they are to avoid cruft.

    If anything, the Silverlight team seems to be the one making abstract arguments. e.g. these moves "simplify" the programming model. I don't understand how violating the Interface Segregation Principle simplifies the programming model or makes code that is easier to test.

    By Blogger John "Z-Bo" Zabroski, at July 16, 2008 at 9:54 AM  

  • I found this Microsoft Connect problem statement registered: https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=333779

    I voted it a 5.

    By Blogger John "Z-Bo" Zabroski, at July 16, 2008 at 6:22 PM  

  • I have noticed that Silverlight doesn't render frames when the browser is minimized. I have been

    told that this is due to optimizations in Silverlight.

    Is there a way to get the rendering going when the browser is minimized or when the silverlight player is launched on a machine with no display?



    By Anonymous Ethen, at October 27, 2009 at 12:51 AM  

Post a Comment | Home | Inference: my personal blog