Platform Rendering Work Weeks in Taipei

The Platform Rendering team is back in Taiwan this month. This time, we’ve split the team across 3 work weeks (Media, Layout, and Graphics.) This lets us focus on the issues specific to each team but still allow for free time to hack and socialize.

Here’s the week’s agenda for the Platform Media team:

Monday
10th March
Tuesday
11th March
Wednesday
12th March
Thursday
13th March
Friday
14th March
10:00 – 10:55 Introduction / Lightning talks
Anthony Jones
Platform Decoder Modules, MP4
Chris Pearce
Media Stream Graph Refactoring
Paul Adenot
Stingray – Introduction and demo of TV
Ide
Content enablement session
Chris / Ide
11:00 – 11:55 Beginner’s guide to <video> code
Chris Pearce
“MediaCodec for playback”
Bruce Sun
“Video Codec Resource
Management”
Blake Wu
“Adjust priority/nice value for media playback threads”
Star Cheng
Testing
JW Wang
Stingray – Introduction of hardware
Nakajima
Shumway (Mozilla Flash Player)
Jet Villegas
Lunch
13:00 – 13:55
14:00 – 14:55 Process reflection
Anthony Jones
Async Decoder Changes
Chris Pearce
Daala
Tim Terriberry
Stringray – AV related achetecture
Yoshioka
TaskTracer Demo
Shelly Lin
4th floor, Mozspace
15:00 – 15:55 Pre-Discussion about Panasonic TV
Shelly Lin
New Recording Features
John Lin
Video Recording
Randy Lin
Seccomp etc
Paul Terriault
Stringray – GFX related achetecture
Ohtani
Hell Kitty
Chris Pearce
16:00 : 16:55 Manager Session
Anthony Jones
Chaos mode debugging and rr
Robert O’Callahan
Media Source Extensions
Chris Double
Our Proposals for Panasonic TV
Shelly Lin
SUPERFLUOUS MEETING #25
Chris Pearce
19:00 – Team dinner @ Din-Tai-Fung, Taipei 101

Here’s the week’s agenda for the Platform Layout team:

Monday Tuesday Wednesday Thursday Friday
9am SVG buffered-rendering (small group) Elephant Mountain walk
10am lightning talks (jet) Shipping CSS Variables (heycam) Vertical Text (smontagu) OMTA (birtles)
11am position:sticky (kip) CSS scroll snapping (roc) Platform Priorities (jet) CSS flexbox (dholbert)
Noon Lunch Lunch Lunch Lunch Lunch
1pm free vsync/refresh driver Platform Priorities (jet)
2pm Intro to Layout (roc/dbaron) rr & chaos (roc) use of performance tools demo (dbaron) SVG performance work (small group requested by Taiwan based employees) (jwatt) wifi display demo (kenchang)
3pm Selection (roc) competitive & standards landscape (dbaron) XPCOM and String classes (dbaron) (hopefully 4th floor) B2G Layout Discussion (jet)
4pm free Moz2D (small group) Platform Priorities (jet) tbd
5pm Restyle performance patch testing (heycam,vilin,cjku) tbd
6pm Team Dinner tbd

This morning, the Layout team decided to take a walk up to Elephant Mountain near the office. A walk in the park with the Layout team has equal parts of viewing the scenery, and discussing rendering issues for Chinese text (both horizontal and vertical!) It was quite fun.

jet_tweaked_small

I’ll post the Platform Graphics team agenda in two weeks after they’re in Taiwan for their work week. Apologies for the use of acronyms and unfamiliar jargon as the Rendering teams often use these terms to describe complex features. Send me a comment if I can help clarify anything.

The Taiwan teams have been very gracious and welcoming as always. See you all again soon!

Planet Mozilla Rocks!

My last two posts didn’t make it on to planet.mozilla.org due to a recent junglecode server move. I asked Robert Accettura to update the entries in the planet database so that server-side redirects aren’t necessary. This post is just testing those changes. It’s also a plug for Planet Mozilla, a most excellent public resource for the Mozilla community.

If you missed these two posts on planet, here you go:
Software Project Estimates: Go Fly a Kite!
Mozilla Platform Rendering in Asia

Software Project Estimates: Go Fly a Kite!

Building Open Source software presents challenges to engineers and managers that differ greatly from proprietary systems. Open Source doesn’t eliminate the need for proper estimates and the predictability of costs that accurate estimates bring. Open Specifications like the W3C web standards add additional complexity to the estimation process. How are you supposed to know how long a project will take, when the specifications are frequently modified by authors and editors who aren’t under your control?

An estimation method that I often use at Mozilla is based on the Niagara Falls Suspension Bridge project. The Niagara Falls Suspension Bridge was the world’s first working railway suspension bridge. It was over 800 feet long, which was a length that was difficult to accurately measure back in 1847, before bridge construction started. The rapid waters and 225-foot high shear cliffs made a direct crossing impossible. The bridge construction must begin by stretching the first wire across the violent stream.

The Niagara Falls Bridge project shares many similarities with a complex software project: multiple stakeholders, unknown scope, dangerous obstacles and difficult terrain. How do you start when you can’t see how far you need to go? In January 1848, fifteen-year-old Homan Walsh flew a kite from the Canadian side of the river and was able to crash the kite into a tree on the New York side. The kite string was used to pull a thicker line back to Canada, then a rope, then metal wire, and construction progressed until a temporary suspension bridge could be built.

The truth is that Homan had a very difficult time getting the kite to fly to the other side! Weather and other unexpected problems left him stranded on the Canadian side for eight days and several failed attempts. Despite the difficulties, his initial kite was essential to the larger goal. When I first talk to engineers about a new software project, I often get estimates that sound like “two or three months, if we’re lucky.” When I hear that, I ask them to go fly a kite.

A “Kite” is just enough code to learn just enough new information about the project so that intelligent estimates can be made. I should note that this code can be very flimsy (like a kite) but functional enough to illuminate the unknown. It’s important that the engineer understands that we will not be shipping the kite! The goal with the kite exercise is to come with estimates that are in multiple pieces of work that each span days or weeks. Estimates that are still measured in weeks may require more kites to break down into days.

Getting that first kite across the gorge can be a very exhilarating experience, and may be just enough motivation to continue the difficult path forward. As an example, a big bridge project for us is Vertical Text and the first kite looked like this. A second kite looked like that.

Mozilla Platform Rendering in Asia

I’ve spent the last 3 weeks traveling in Asia for Mozilla and W3C business. On May 20 to 24, my team was in Taiwan for a series of meetings on Platform Rendering (Layout, Graphics, and Media.) It’s always an enlightening experience for me when I spend time with such a talented group of people:

The Mozilla Firefox Platform Rendering Team on location in Taiwan.

Going to Taiwan was a very special event. The local Taiwan office was awesome, and the local staff was very wonderful, taking great care of us while we were there. Beautiful and fast rendering requires a deep understanding of the underlying hardware, and if you’re serious about hardware, you eventually have to go to Taiwan. Meeting and working with the local Taiwan team was great. They’re so smart, and eager to learn about the Firefox Platform. We spent a lot of time getting them caught up on Rendering infrastructure, and various Mozilla-specific topics (Open Source, Open Specs, Code Review, etc.)

I’ve pasted the calendar we followed for the week, below. The links point to raw notes from the sessions (where available.) I apologize for the lack of context in some notes, as they’re meant for the attendees. They’re in this blog post to give you an idea of what the Rendering Team does when we get together in a large group every few months.

Taiwan Rendering Sessions

Monday Tuesday Wednesday Thursday Friday
9am Lightning talks Graphics: SkiaGL Canvas (snorp) Graphics:WebGL (Vlad) Layout/Media: Implementing WebVTT CSS features (rillian/dbaron)
10am Free Graphics: SkiaGL Content (Jeff M.)
Layout: Intro to Style System (dbaron)
Graphics: WebGL (Vlad)
Media: libCubeb/Audio Latency (Kinetik)
Platform:Cycle Collection 101 (khuey) Graphics: Timing Attacks (bjacob)
Layout:Layout Documentation – Conf Room B (jwir3)Media: ???
11am Free Graphics: Async Canvas (Vlad, snorp)
Layout: Intro to Dynamic Changes in Layout (dbaron)
Graphics: WebGL (Vlad)
Media: Web Audio (padenot)
Layout: CSS Masking
Media: Fixing MediaStreamGraph video propagation (roc)
Graphics: Imagelib (joe)
Other: WebGL (Vlad)
Layout: ???
Media: ???
Noon Lunch Lunch Lunch Lunch Lunch
1pm Graphics: OMT-Compositing (Nical – “D”)
Layout: Layout 101 (Elika – “C”)
Layout: APZC Graphics: Moz2D/Player2D (Bas) Layout:Layout fuzzer status (multicol security bugs, etc.) – Conf Rm. A Graphics:Intro to Gecko Graphics (mwoodrow)
Layout: ???
Media: ???
2pm Graphics: Layers and buffers (bjacob) Layout:Web Animations (birtles)
Media: MediaSource Extensions (kinetik)
Graphics: OMT Canvas Layout: Web Animations contd. (birtles) Graphics: ???
Layout: Standardization and Mozilla (dbaron/abr) «IETF Slides»
Media: ???
3pm Graphics: Gralloc
Layout:CSS Variables / CSS Cascade (heycam, dbaron – “C”)
Layout: CSS Graphics Graphics: OMT Animation
Layout: (see graphics)
Media: Media Decoding (cdouble) OMX Codec (sotaro)
Graphics: B2G/Android Graphics Testing
Layout:CSS Writing Modes – Vertical Text (fantasai)
Graphics: ???
Layout:Layout Performance (jet)
Media: ???
4pm Media: WebVTT/TextTrack (rillian) The Mozilla Way (roc) Graphics: CSS Filters
Layout:CSS3 Fonts
Code Reviews – The Mozilla Way (roc) Work Week Wrap-up (jet)

The Shumway Open SWF Runtime Project

Last year, I wrote a blog post regarding Flash that generated a fair bit of discussion. Since then, the Research team at Mozilla has been working on solutions: https://blog.mozilla.org/research/2012/11/12/introducing-the-shumway-open-swf-runtime-project/
As the research blog states, “…Shumway is very experimental, is missing features, contains many defects, and is evolving rapidly.” If you’re a technical person who wants to help the current situation with Flash and the Open Web, please help us out.

 

 

 

Graphite Smart Fonts in Firefox

The Gecko Layout Engine added Graphite Smart Fonts support starting in Firefox 11. The Mozilla Firefox documentation for Graphite is still under construction but already contains lots of useful information for Web Designers and Developers.

The number of people who will benefit from Graphite is the small subset of the population that reads and writes with lesser-known scripts that require complex ligatures and rules for glyph substitution. In other words, we expect this work will appeal to a rather small underrepresented percentage of the web who communicate using uncommon writing systems. Firefox fully supports OpenType for the majority of writing systems on the web.

The key benefit of Graphite is that it allows type designers to build in complex script handling logic into fonts themselves, rather than relying on an application library like OpenType to explicitly implement support for it.  That’s why it’s so useful for display and layout of scripts that require features not available in OpenType.  There’s a lot of overlap with OpenType and the format itself simply adds extra data to OpenType fonts (the glyph and metric data is the same).

Graphite allows fonts to embed rich programs that implement the complex rules that govern glyph behavior in relation to other glyphs in a text run. This rich API presents challenges in an application as widely-distributed as Firefox, especially in regards to security and integration with the rest of the text subsystems.  We’re currently testing, validating and documenting the Graphite feature in Firefox. We’re interested in getting feedback from the Font Developer community to ensure that Graphite is expressive enough for complex writing systems. If you have experience with OpenType layout, we’d like to see similar layout functionality implemented using Graphite. We’re looking for examples that could include non-Latin script support (e.g. Arabic, Thai, Indic scripts) or Latin/Cyrillic/Greek fonts with extensive feature support.  A symbolic “WingDings” font with complex layout rules could also be a useful example. Please reply in the blog comments if you are interested and available to help us test and verify Graphite’s Layout API’s.

I’m posting about Graphite because it’s another example of why working on Firefox is so different from working on anything else. We don’t optimize our actions to generate maximum revenue, we do things because we want to give the entire planet equal and open access to the web. Proper text layout and rendering for complex scripts enables free expression across communities who would otherwise not have a global voice. We think it’s well worth it in service of the greater good for an Open Web.

Toggle Paint Flashing for Firefox

Paint Flashing is now in the official Release channel starting in Firefox 11.  I posted a new add-on that let’s you enable Paint Flashing with one click: https://addons.mozilla.org/en-US/firefox/addon/toggle-paint-flashing/

68828

This tool is useful for quickly inspecting how Firefox renders your web pages as it visually indicates how much screen area is being painted after the page layout is computed.The source code for the add-on is also posted.

 

 

How I got started hacking on Firefox

One of the Gecko Layout & Rendering team’s main responsibilities is the continuing development of CSS in Firefox. I recently modified the CSS style system to allow nested rule parsing. This bug fix taught me a lot about the CSS parser and how styles cascade through the rest of the Layout engine. It took me a little while to set up a dev environment, understand the bug, write tests, get the code written, reviewed, and checked in. I now have a much better understanding of what it takes to move code through the Mozilla source trunk.

I suspect that there are other programmers out there currently lurking around the Mozilla community, intimidated by the scale of the source tree, and wondering where to start hacking on Firefox. I highly recommend starting with the Gecko Overview started by L.David Baron, Mozilla Principal Software Engineer, to help beginners understand the browser engine. Thanks, David!

Firefox Invalidation

As promised, I’m going to call out Firefox Layout & Rendering code changes that my team has been working on. This bug fix from Bas Schouten (as reviewed by Robert O’Callahan) was one of those light-bulb code changes that have really advanced my understanding of the Gecko Layout Engine’s graphics code:

https://bugzilla.mozilla.org/attachment.cgi?id=574190&action=diff

This check-in implements “Paint Flashing”, a diagnostic tool that shows when the browser is invalidating (or repainting) a screen region that has changed. It’s very useful for diagnosing redundant (painting too often) or greedy (painting too large an area) invalidation. Both kinds of bugs are notoriously hard to find.

Paint Flashing will paint a random color (at 20% opacity) to indicate the “damaged” area of the screen that the browser is repainting. You can enable Paint Flashing by setting this preference to true:

nglayout.debug.paint_flashing

If you’re running a DEBUG build, you can get to it in the Layout Debugger. The code is deceptively simple but shows the logical bottleneck where invalidation occurs in Firefox. We’re already using this tool as we evaluate larger invalidation changes that aim to dramatically speed up rendering in Firefox. You can also use this tool as you develop your web apps to see if performance issues are the result of invalidation problems. You can tune your app much better if you know how much screen area is getting painted from your javascript or CSS code.

Flash: I’m not dead yet!

My last post generated a fair bit of discussion around the true fate of Flash within Adobe. The original announcement from Adobe and the associated staff reductions had caused a lot of speculation within the Flash community. I should clarify some of my statements, especially the parts about the Flash Pro authoring tool and Flash Player.

All software follows a similar life cycle from 1.0 to a final version when it effectively becomes frozen in time. I didn’t mean to imply that Adobe is pulling Flash Pro from store shelves like salmonella-tainted spinach. A familiar pattern is to move software engineering to a low-cost geography like India or China as growth slows. This is not a bad thing, and actually lets the software have a long tail of maintenance and incremental feature work at a cost that allows it. The consumer can still count on that software being available for some time with support (usually from the same low-cost geography.) A handful of engineers will remain in the original geography to consult the new engineers as the hand-off occurs. The Flash Pro team in the US will have a handful of engineers left behind to manage the hand-off so, technically, the entire team is not getting eliminated at this time. Over time, the idea is to have your higher-cost engineers migrate over to your higher-growth projects. This is what occurred many years ago at Macromedia when Director moved offshore and the team moved over to working on Flash.

Last Tuesday’s announcements did not follow this pattern. Instead of moving the mobile Flash Player to a low-cost geography, it was discontinued outright. Instead of moving many affected Flash Platform engineers to the next high-growth agenda, they’re getting sent home. These events are what prompted me to write my last post as my friends at Adobe were blindsided by the whole affair.

I have a great deal of respect for the Adobe product managers who are trying to contain the damage and reverse the Elop Effect from the original announcement. It does seem like they were not made aware of the Tuesday announcement in advance. I sincerely hope they will be successful at affecting change within Adobe to keep Flash around for a good while longer.