{"id":46,"date":"2009-09-17T23:02:59","date_gmt":"2009-09-18T06:02:59","guid":{"rendered":"http:\/\/gameangst.com\/?p=46"},"modified":"2009-09-17T23:02:08","modified_gmt":"2009-09-18T06:02:08","slug":"minimizing-code-bloat-for-faster-builds-and-smaller-executables","status":"publish","type":"post","link":"http:\/\/gameangst.com\/?p=46","title":{"rendered":"Minimizing Code Bloat for Faster Builds and Smaller Executables"},"content":{"rendered":"

No matter how much code you have, somebody always wants more. \u00a0That’s certainly been the case with every game and every engine I’ve worked on. \u00a0My good friend Kyle<\/a>, who has for years been tracking all sorts of wonderful statistics about the development of Despair Engine<\/a>, recently announced that we’d surpassed 1.5 million lines of principle engine code. \u00a0That doesn’t count build systems, tools, tests, middleware, or scripts; that’s 100% Despair-original C++ code the compiler has to wade through every time we build the game.<\/p>\n

As if the code we write weren’t bad enough, our engine is particularly adept at generating code<\/a>. \u00a0Those 1.5 MLOC result in a hefty 37 megabyte executable on the Xbox 360. \u00a0Kyle estimates that we’ll exceed 2 million lines of code before the end of this console generation. \u00a0We already have an executable that can’t fit into main memory on a PS2 or a Gamecube. \u00a0By 2014 it won’t fit onto an Xbox or Wii either. \u00a0I don’t think we’re in danger of exceeding the capacity of a PS3 or Xbox 360, but the prospect is enough to keep me up at nights.<\/p>\n

Looking back at the graphs of our executable size and build and link times, I’m proud to see a few dips punctuating the otherwise consistent upward march. \u00a0At least a few of those dips correspond to weeks when I turned my sleepless nights into successful assaults on code bloat. \u00a0What’s particularly interesting about those dips is that they don’t correspond to any drops in our code line count. \u00a0In my biggest success I reduced the size of our executable by 25% or over 7 megabytes. \u00a0At the same time I cut our link times by 19%, and yet I did it all without removing a single line of code! \u00a0How?<\/p>\n

The first place to start when trimming a codebase is figuring out where to look. \u00a0In an engine of 1.5 million lines of code there are a lot of places to hide. \u00a0My strategy when dealing with problems like this is to gather as much data as possible into a format that is amenable to analysis. \u00a0For code this means extracting symbols from the compiled binaries along with their size, type, source file, and any other details you can get your hands on. \u00a0Once you have a dump of all the symbols in your compiled code, it is a pretty simple matter to write a script to collate it and generate whatever statistics you find useful.<\/p>\n

There are two methods I use for extracting data from compiled code. \u00a0The first is a\u00a0quick and handy utility called DumpBin. \u00a0DumpBin<\/a> is part of the Visual Studio toolchain and is available for all Microsoft targets. \u00a0There are similar tools available for every compiler I’ve encountered. \u00a0The gcc equivalents, for example, are objdump<\/a> and nm<\/a>. \u00a0These tools are useful because they can work with just about any compiled binary. \u00a0They target standard binary formats like PE\/COFF<\/a> and ELF<\/a>, and consequently they work on object files, library files, executables and DLLs. \u00a0The ability to extract information from intermediate files in the build process is essential if you’re worried as much about build times as you are about final memory usage.<\/p>\n

My preferred way to use DumpBin is to specify the \/headers<\/em> option to extract COMDATs from object files compiled with the \/Gy<\/a> option. \u00a0For example:<\/p>\n

for \/r . %n in (*.obj) do DumpBin \/headers “%n” >> all_headers.txt<\/em><\/p>\n

<\/em>A portion of the output from such a dump looks like this:<\/p>\n

SECTION HEADER #14E\r\n   .text name\r\n       0 physical address\r\n       0 virtual address\r\n       7 size of raw data\r\n   22D6B file pointer to raw data (00022D6B to 00022D71)\r\n   22D72 file pointer to relocation table\r\n       0 file pointer to line numbers\r\n       1 number of relocations\r\n       0 number of line numbers\r\n60501020 flags\r\n         Code\r\n         COMDAT; sym= \"public: virtual __thiscall Despair::Reflection::Object::~Object(void)\" (??1Object@Reflection@Despair@@UAE@XZ)\r\n         16 byte align\r\n         Execute Read\r\n\r\nSECTION HEADER #14F\r\n   .text name\r\n       0 physical address\r\n       0 virtual address\r\n       5 size of raw data\r\n   230B1 file pointer to raw data (000230B1 to 000230B5)\r\n       0 file pointer to relocation table\r\n       0 file pointer to line numbers\r\n       0 number of relocations\r\n       0 number of line numbers\r\n60501020 flags\r\n         Code\r\n         COMDAT; sym= \"public: int __thiscall Despair::BroadcasterReceipt::AddRef(void)\" (?AddRef@BroadcasterReceipt@Despair@@QAEHXZ)\r\n         16 byte align\r\n         Execute Read<\/pre>\n
As you can see, the result is an easily parsable list of packaged functions with symbol names and sizes clearly specified. \u00a0As great as the \/headers <\/em>dump is, it isn’t perfect. \u00a0Data symbols aren’t packaged as COMDATs and although DumpBin can extract data from executables, executables don’t contain COMDATs so there isn’t much to see.<\/p>\n
To analyze the contents of an executable, a second approach is needed. \u00a0My solution is to reference the debugging information provided by the linker. \u00a0With Microsoft’s tools, debugging information is stored in a PDB<\/a> file that accompanies the executable. \u00a0Similar to DumpBin’s \/headers<\/em> dump, the PDB contains information about symbols in the compiled binary. \u00a0Unlike the \/headers<\/em> dump, however, the PDB file contains data as well as code symbols, and the information it contains is post-linker optimization which is important if you’re more worried about memory consumption than compile and link times.<\/p>\n
There aren’t any standard tools for dumping the contents of a PDB, but there is a DLL provided with the Microsoft toolchain that makes it pretty easy to write your own. \u00a0I’ve used the Debug Interface Access SDK<\/a> included with Microsoft Visual Studio to plumb the contents of an executable and generate symbol tables that can be analyzed alongside a COMDAT dump.<\/p>\n
In an upcoming series of articles I’m going to explore the most common causes of C++ code bloat and take a closer look at how you can use DumpBin and the PDB to identify and remove the worst offenders from your code. \u00a0When I’m done I’ll share some sample code which you can use to assess the level of code bloat in your own engines.<\/p>\n
First up, template overspecialization<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
No matter how much code you have, somebody always wants more. \u00a0That’s certainly been the case with every game and every engine I’ve worked on. \u00a0My good friend Kyle, who has for years been tracking all sorts of wonderful statistics about the development of Despair Engine, recently announced that we’d surpassed 1.5 million lines of […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[8,14,11,7],"_links":{"self":[{"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/posts\/46"}],"collection":[{"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/gameangst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=46"}],"version-history":[{"count":24,"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/posts\/46\/revisions"}],"predecessor-version":[{"id":278,"href":"http:\/\/gameangst.com\/index.php?rest_route=\/wp\/v2\/posts\/46\/revisions\/278"}],"wp:attachment":[{"href":"http:\/\/gameangst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=46"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/gameangst.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=46"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/gameangst.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=46"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}