When developing systems or devices based on embedded Linux or Android, does it make sense to use commercial development tools? In this guest column, Brad Dixon, Director of Open Source Solutions at Mentor Graphics, suggests several reasons why commercial development tools and support can potentially save time, resources, money, and opportunity costs.

 

Embedded Linux and its popular offspring Android have reshaped embedded system development over the past decade. The impact of open source software is unmistakable. A quick look at the available technologies reveals that at every level of abstraction, including tools, silicon, and even at the application level, there is an abundance of technical expertise, which when applied correctly can benefit developers and managers of next-generation open source products.

Open source-based development tools offer a solid foundation for developers and have evolved rapidly over the past 15 years, powering the explosive growth of Linux, Android, and most web and cloud technologies. The GNU Project, founded and supported by the Free Software Foundation (FSF), is arguably one of the most influential projects in the Free Software movement and has today become the cornerstone upon which hundreds of millions of lines of code depend. The grandfather of GNU tools is the venerable GCC (GNU Compiler Collection) which is most commonly, but not always, used with GNU Binutils (linker, assembler, and other tools), and GDB (GNU Project Debugger).

The good news is that today’s embedded software developer is surrounded by countless numbers of technologies and innovation — courtesy of various open source communities. The bad news is that even with all of this great technology, there are still plenty of shortcomings in the overall tool flow that may limit a developer’s success. These shortcomings are often most visible when proprietary tools are employed and development teams spend more time troubleshooting their toolchain rather than focusing their time and effort on activities that add commercial value to their product.
 

Some advantages of proprietary tools

Proprietary developer tools are both successful and valuable to developers in a way that cannot be dismissed casually. Embedded software developers generally are pragmatic and favor outcome over ideology. For many core toolchain technologies, open source tools have become the defacto standard by merit of the powerful capabilities and broad device support offered. In general, proprietary tools thrive in specialty niches whereas open source tools quickly expand to support broadly desired functionalities. In many cases, commercial firms selling complementary products have sponsored the advancement of open source tools.

However, before initiating any type of proprietary toolchain development, careful consideration should be given to two common areas of risk management:

• Integration risk management

  Almost all project schedules that depend upon open source development tools plan for some investment of time to knit the individual open source technologies into a coherent developer workflow. For embedded software projects, this typically includes the preparation of cross-compilation and debugging tools, and integration with an IDE such as Eclipse, Emacs, or vi. While the availability of these assistive technologies might at first seem beneficial, managers often mistakenly accept the notion that a “build from source” mechanism mitigates integration risk.

  Interestingly, projects don’t typically get stuck when a team is trying to figure out how to build their cross-development tools and libraries. They most typically get stuck when the tool flow fails to work in some subtle, unanticipated way.

  To mitigate these risks, management needs to accept that a “build from source” methodology does not necessarily mean that the engineering team has internalized all of the necessary expertise to do the job correctly. Managers should consider designating community liaisons who invest in monitoring the various open source communities they depend upon and build social relationships with other developers. Larger organizations should consider forming a dedicated tools integration, delivery, and maintenance organization.

• Schedule risk management

  Toolchain defects can stop a project dead in its tracks. When a defect in a core system library, linker, assembler, or cross-compiler strikes, timely resolution by an expert is essential. Unfortunately, software defects rarely strike in predictable locations and those responsible for maintaining toolchains need to develop expertise at every level of the toolchain and library software stack. All too often, this means developing competency in a frighteningly large swath of code.

  In addition to large amounts of code, community projects can also change quickly. Consider just one central technology such as the GNU Compiler Collection. Ten years ago the GNU Compiler Collection had only about two million lines of code. Over the past ten years, 300,000 lines of code on average have been added to the GNU project annually. Add in normal modifications and the codebase turnover is staggering.

  This is important to mention because compiler defects are most typically detected deep in the engineering schedule when projects can least afford the delay. By this time, the open source community has significantly modified their codebase and are often interested in diagnosing and fixing defects only at the tip of their development branch. There simply may not be much interest in solving a problem with an old version of the software that the engineering team is now dependent upon.

  This situation should illustrate that the common “download, fork, and forget” approach for creating a developer workflow is at best short-sighted. Developers need to either stay relatively current with new tool versions and community evolutions (bearing frequent integration costs) or make provisions to staff a team to maintain a private branch for the entire product lifecycle.

What about commercial tools?

Managers who support the use of open source development tools can reap numerous advantages turning to a commercial development tools product such as Mentor Embedded Sourcery CodeBench, from Mentor Graphics, illustrated below.

Example: Mentor Embedded Sourcery CodeBench
(click image to enlarge)

A commercial partner can potentially benefit a project by reducing integration risk through the use of an engineered, tested, and supported product based on open source technologies. Further, a commercial partner’s engineering staff can reduce the burden of a customer having to keep up to date with the progress of community software projects. For example, when a suspected tool bug is spotted, the commercial tool product’s own engineering staff is generally prepared and ready to diagnose and fix the defects. Additionally, commercial developer tool partners often supplement the open source workflow with proprietary components that integrate independent tools (such as GDB and JTAG probes) or enhance the developer experience through plugins, analysis tools, or agents simply not available from community projects.
 

When free software is anything but free

As open source software continues its widespread acceptance, managers and senior developers must be aware of the challenges and trade-offs of Free Software. For example, a senior manager of development has a small team of six developers; one senior technical lead and five mid-level developers. The lead developer might spend 20 percent of his time, on an annual basis, creating and maintaining the tools he’s built for the team and chasing down bugs with the help of various open source communities.

Suddenly, the team encounters a compiler bug that stalls the team for a week. After a day or so, the five developers move to other tasks, but the lead developer spends a week diagnosing and fixing the issue. Toss in the daily meetings with the senior manager and the entire team on this critical, late-cycle defect is now wasting valuable time and money. The company could easily burn through $25,000 of the lead developer’s time on an annual basis creating, maintaining, and fixing the tools when an outside commercial tool flow, even at minimal cost (say $5,000) would in the long run be more prudent. Not only would the commercial option cost less, but there’s also the potential lost market opportunity to be considered.
 

And the answer is…

The impact of open source software on embedded development projects has far-reaching implications. To many developers, open source software appears to be a panacea for complex challenges in embedded system development. However, closer scrutiny reveals the potential for challenging and possibly costly pitfalls to beset even the most astute software developer. In most cases, developers would be better off leveraging a commercial toolchain than trying to build an open source tool flow themselves. And while the benefit of Free Software may seem attractive, one simple compiler bug can potentially negate the anticipated savings.

Commercial tool flow vendors almost always includes the latest enhancements to the GNU tool chain; quality assurance is virtually guaranteed by running hundreds of thousands of tests in its own hardware lab facilities. Some companies do exceedingly well by building their own GNU tool chain. These companies however, have learned that a constant interface with the open source community is a must if any type of open source development endeavor is to succeed.
 

About the author: Brad Dixon is the Director of Open Source Solutions at Mentor Graphics, responsible for open source developer tools and Linux. Brad has been at the nexus of open source and embedded software as a developer, FAE, and product manager since 2000.