Class Reflection in the Despair Engine

Reflection is, without a doubt, my favorite computer programming language feature.  It is unfortunate therefore that I’ve spent most of my professional life programming in C++, a language that has stubbornly refused to adopt a mechanism for class reflection.  There is a reason why I’ve stuck with C++ however; it is the highest level language available today that still provides unobstructed access to the hardware platform on which it sits.  Whatever its faults, it never prevents me doing what I have to do.

When Kyle Wilson and I were laying out the design for Day 1’s Despair Engine, low-level support for reflection was a must-have on the list of engine features.  We were designing the largest, most ambitious engine either of us had ever worked on, and we didn’t want it to be clogged with thousands of lines of boilerplate code.  We wanted a simple, easy to use mechanism by which common libraries for disk serialization, UI generation, network synchronization, script integration, and debugging could operate intelligently on a diverse field of heterogeneous objects.  We’d both been down the route of virtual “Save” and “Load” calls implemented on every object as well as the route of UIVehicle and NetVehicle proxy objects, and we were intent on finding a better way.  At the same time, however, we were keenly aware of the dangers of over-generalizing.

We knew that the engine we were designing would have to grow to support dozens of engineers and multiple simultaneous projects, and that it would have to be flexible enough to adapt to the unpredictable requirements of future games and hardware.  Taking all this into consideration, here is the list of goals we drafted for our reflection system:

  • Zero reliance on custom compiler tools or non-standard compiler extensions.  100% portability was essential because we expected to support platforms whose compilers hadn’t even been written yet.
  • Zero reliance on RTTI or exceptions.  Both of these features come with a cost and are regularly disabled or broken on consoles.
  • No header pollution.  Reflection definitions had to be confined to cpp files so that compile time costs could be minimized and so that modifying a reflection definition didn’t require a full rebuild.
  • Minimal reliance on macros.  Provide type safety and lexical scoping whenever possible.
  • Zero reliance on external data.  Reflection metadata had to be available at static initialization time, tightly bound to class definitions.
  • Efficient support for native types like __m128 and custom low-level types like struct Vector2 { };
  • Support for sophisticated but non-polymorphic types like standard library containers.
  • Support for polymorphic classes and multiple inheritance of abstract interfaces.  We use a COM-like architecture in our engine which clearly separates objects from interfaces.  We deliberately chose not to tackle virtual inheritance with this system.
  • External extensibility.  Reflection definitions had to support metadata specific to systems that the reflection library itself had no knowledge of.  Adding new types or attributes to the system shouldn’t require a massive rebuild.
  • Support for overriding or circumventing the system for special cases.

In addition to these general requirements for the system, we had several specific uses for reflection already in mind which added their own requirements:

  • Disk serialization using the reflection system had to support backward and, to some degree, forward compatibility.
  • Network synchronization using the reflection system had to be able to operate on a subset of the class data and to efficiently extract deltas from class state.
  • UI generation from reflection definitions had to provide context-specific as well as type-specific widgets.  A “radius” property for a volume trigger, for example, should be able to provide a 3-D visualization as well as a range-limited edit box, context-sensitive help, and notifications to other UI elements when it changes.
  • Scripting languages like Lua and Python had to be able to dynamically bind to objects using their reflection definitions, but once bound they should be able to operate on the objects efficiently without requiring table lookups or string comparisons.

It fell on my shoulders to provide the initial design and implementation of Despair’s reflection library, dsReflection.  Luckily reflection is nothing new to games, so I had a lot of inspiration to draw upon.  One of my favorite early implementations of reflection in a game engine is in the Nebula Device.  The Nebula Device’s approach to reflection was fairly straightforward–hard coded companion functions for every object populated tables with metadata like command names and function pointers–but it was used to great effect to expose a great deal of the engine to Tcl and from there to perform serialization and network synchronization.

Although the Nebula Device was a great proof-of-concept for me, it didn’t meet many of the requirements Kyle and I had put forth.  It was very limited in the kind of data it could reflect and syntactically it was messy and required a lot of boilerplate code.  The inspiration for the syntax of Despair’s reflection definitions came instead from C# and luabind.  We hoped to be writing a lot of tool and support code for the engine in C# so it made sense to try to match its syntax, and luabind is an ingenious example of how C++ compilers can be manipulated into parsing very unC++like code.

What I ended up with is an embedded, domain-specific language implemented through a combination of clever operator overloading and template metaprogramming.  Many of the core concepts in dsReflection are enabled by features of Boost’s metaprogramming tools.  The reflection definition language is used to construct metadata for classes.  The metadata holds references to fields and methods of a class.  Fields can be raw member pointers as well as combinations of getters and setters.  Methods can be member functions with variable numbers of parameters of mixed types.  Methods can expose concrete, virtual, and even pure virtual functions.  All reflected elements can be tagged with user-defined properties similar to Attributes in C#.  These properties are how external systems annotate classes, fields, and methods with information specific to their needs.  Reflection definitions can be applied to non-polymorphic classes, but to take advantage of the full feature set reflected classes must derive from Reflection::Object which introduces a single virtual function.  In its initial implementation dsReflection supported global and file static variables and functions through reflection namespace definitions.  This capability still exists, but we’ve found little use for it and I’ve often considered removing it entirely.

At run time the reflection system provides a wide range of services.  It allows for RTTI-like queries for comparing object types and inheritance hierarchies and it allows for type-safe dynamic casts between reflection objects.  Class definitions for objects can be inspected at run time, and their component elements can be operated on in a variety of ways.  Fields can be read from and written to, and they can also be bound to strongly-typed, templatized wrapper objects called Reflection::Variable<T>.  Similarly, methods can be called with parameters provided through an abstract interface or they can be bound directly to boost::function objects.  Property objects can be extracted from all reflected elements and the properties themselves are reflection objects so they support the same rich level of introspection.

The Despair Engine is about four years old now and it has the battle scars to prove it.  The reflection library is one of the foundational elements of the engine, and it has held up remarkably well under the strain.  As we hoped, dsReflection is leveraged to support all the systems describe above, as well as several others that we never anticipated.  It is the kind of feature that once you have it, you can’t imagine life without it.

That said, there have been some struggles along the way.  The biggest struggle with the reflection library has been keeping compiler performance and executable code bloat under control.  These were both serious concerns going into the system which we attempted to address in the design.  The system does meet the requirement that class reflection definitions are specified in cpp files and not in headers, but nevertheless the reflection definitions for classes can be big (really big!) and there is a lot for the compiler to wade through.  A bigger problem for us than the compile time has been the amount of code generated by the compiler to instantiate the reflection definitions.  Reflection definitions are built automatically at static initialization time, so run time performance and memory consumption aren’t an issue, but I’ve twice had to revisit the system’s internals to optimize its code gen and bring our link times and final executable size back to within reasonable limits.  Both Sony and Microsoft have shared in this pain; few engines out there put pressure on a linker like ours.

Another struggle with the reflection system has been training and documentation.  It is a powerful and flexible language which looks only vaguely like C++, and learning all the ways in which one can and, perhaps more importantly, should use the system is a challenge.  Perhaps the best decision we made in this regard was to provide a comprehensive set of unit tests for the library.  Unit tests aren’t the best form of documentation, but they can be invaluable at enforcing undocumented and poorly specified behaviors in systems for which stability and backward compatibility are essential.  Our unit tests provide an answer of last resort to questions like, “can I reflect a std::map<enum, object> and have the serialization maintain backward compatibility as enum values are changed, added, and removed?”  The magic unit test says “yes!”

Detailing the exact workings of dsReflection is way beyond the scope of this article, but I can provide a sample reflection definition that gives a pretty clear picture of how the system plays out in practice.  This sample isn’t something that actually exists in any game, but it is a completely valid reflection definition that would compile with our code.  I haven’t tried to hide any of the messiness.  Most of the complexity comes from specifying a pretty sophisticated user interface with custom widgets and data-dependent views, but that’s hardly unusual.  When our components expose a UI, most of the time we’re pushing for more customized, more specialized behavior, even though it makes the reflection definitions tricky.

// DecalObject.h

class IDecalObject
{
    // abstract interfaces can carry reflection definitions
    REFLECTION_CLASS_DECLARE(IDecalObject);
};

class DecalObject : public Reflection::Object, public IDecalObject
{
public:
    const string& GetTextureFilename() const;
    void SetTextureFilenameBlocking(const string& filename);

    enum DecalType
    {
        kDecalType_Static,
        kDecalType_Random,
        kDecalType_Special
    };

    ResultCode StartLoggingDebugStats(const char* filename, bool echoToScreen);
    void StopLoggingDebugStats();

private:
    string GetAspectRatio() const;  // returns aspect ratio of currently loaded texture
    bool IsRandomType() const;      // helper checks if m_decalType is random

    ResourceKey         m_textureResource;
    DecalType           m_decalType;
    float               m_maximumImpactAngle;
    std::vector<Color>  m_randomColorPalette;
    bool                m_applyToSkinnedObjects;

    REFLECTION_CLASS_DECLARE(DecalObject);
};

REFLECTION_ENUM_DECLARE(DecalObject::DecalType);

// DecalObject.cpp

#include "DecalObject.h"

// empty reflection definition for IDecalObject
REFLECTION_CLASS_DEFINE(IDecalObject);

REFLECTION_ENUM_DEFINE(DecalObject::DecalType)
.name("static",     DecalObject::kDecalType_Static)
.name("random",     DecalObject::kDecalType_Random)
[
    Prop::Description("A randomly tinted decal.") // custom help for UI / debug console
]
.name("special",    DecalObject::kDecalType_Special)
[
    Prop::EditEnumPrivate() // code only, not to be exposed to UI
]
;

REFLECTION_CLASS_DEFINE(DecalObject)
[
    Prop::GroupProperty("Graphics") +
    Prop::FriendlyName("Decal Component") +
    Prop::Description("Creates a decal at the current location.")
]
.field("textureResource", &DecalObject::m_textureResource)
[   // direct serialization of a custom ResourceKey type
    Prop::Save() + Prop::Copy()
]
.field("textureFilename", &DecalObject::GetTextureFilename,
                          &DecalObject::SetTextureFilenameBlocking)
[   // redundant exposure of m_textureResource but through more UI convenient,
    // blocking member functions
    Prop::FriendlyName("Texture") +
    Prop::Description(
    "Specifies the texture that will be applied by the decal.  The aspect ratio of the "
    "texture will define the aspect ratio of the decal.") +
    Prop::EditResourcePicker("tif;png")
]
.field("aspectRatio", &DecalObject::GetAspectRatio)
[   // read-only exposure of a fake, UI only field
    Prop::FriendlyName("Aspect Ratio") +
    Prop::Description(
    "The aspect ratio of the decal.  This is determined by the texture selected for "
    "the decal.") +
    Prop::EditText()
]
.field("decalType", &DecalObject::m_decalType)
[
    Prop::FriendlyName("Type") +
    Prop::EditEnumCombo<DecalObject::DecalType>() + // provide a combo box tailored to
                                                    // this enum
    Prop::Save() + Prop::Copy()
]
.field("maxImpactAngle", &DecalObject::m_maximumImpactAngle)
[
    Prop::FriendlyName("Maximum Impact Angle") +
    Prop::EditAngle() + // field holds radians, UI presents as degrees
    Prop::Save() + Prop::Copy()
]
.field("paletteSize",
       Reflection::GetSize(&DecalObject::m_randomColorPalette),
       Reflection::SetSize(&DecalObject::m_randomColorPalette))
[   // Helpers expose a std::vector as resizeable within the UI. Fancier definitions can
    // provide custom dialogs for modifying container types
    Prop::FriendlyName("Random Palette Size") +
    Prop::Filter<DecalObject>(&DecalObject::IsRandomType) + // only random decals expose
                                                            // a color palette
    Prop::EditNumeric(0, 100) +
    Prop::Save() + Prop::Copy()
]
.field("palette", &DecalObject::m_randomColorPalette)
[   // Type traits allow the reflection system to operate on templatized standard library
    // and custom container types, including associative containers like std::map
    Prop::FriendlyName("Random Palette") +
    Prop::Filter<DecalObject>(&DecalObject::IsRandomType) +
    Prop::EditContainer( Prop::EditColorPicker(), "Color Entry" ) +  // The container
    // holds a simple type so addition properties are specified for it.  Containers can
    // also hold full reflection objects with their own reflection definitions.
    Prop::Save() + Prop::Copy()
]
#ifdef _DEBUG
// a few fields and methods for debug console
.method("StartLoggingDebugStats", &DecalObject::StartLoggingDebugStats)
[
    Prop::Description(
    "StartLoggingDebugStats log_file_name echo_to_screen\n"
    "  Logs memory and performance stats to a file and, optionally, the screen.")
]
.method("StopLoggingDebugStats", &DecalObject::StopLoggingDebugStats)
.field("applyToSkinnedObjects", &DecalObject::m_applyToSkinnedObjects) // No props, debug
                                                                       // console only
#endif
;

15 Comments

  1. Daniele says:

    Very interesting. I’m thinking about the same thing, but it’s a lot of work… All this would be more simple if boost::langbinding had been completed.

    Daniele

  2. david says:

    So when serializing do you save things like “position”,xyz with the associated string overhead ? a hashed version of the string?

    Or just write the float3 position, and in that case, how can you mantain backwards compatibility ?

    Very interesting read, thank you !

  3. Ruslan Shestopalyuk says:

    Very insightful, thanks for sharing.
    We have quite similar system in our engine, except that we made a tradeoff and dropped the “zero reliance on external data” requirement, so basically things like “friendly name”, “description”, editor type and parametrs are specified externally from the XML file (or, rather, several files, split by the classes domain).

    It turned out to work quite fine, reducing the code bloat, simplifying the c++ side of the reflection, and making it somewhat data-driven (for example, no need to recompile the data to change the range of the slider control, generated by the certain property).

  4. Adrian Stone says:

    David, serialization is handled through an abstract Serializer interface similar to C#. We have serializers for the Windows registry, a text xml file format, a binary file format, and others. In general the serializers store name and value pairs to handle backward compatibility. My original intention was to write a more streamlined serializer to use in the final product when backwards compatibility could be sacrificed, but since all our load-time optimization was done on the production serializer with full backwards compatibility, we decided it was an unnecessary risk / optimization to change formats for the final product.

  5. Adrian Stone says:

    Ruslan, very interesting. A colleague of mine was in favor of storing some of the reflection data externally as it sounds like you have done. I don’t regret my decision to keep all the data in one place, but I do wish I had the opportunity to try the alternative for comparison. It’s good to know it worked out well for you.

  6. Jedd Haberstro says:

    Great post.

    How does method invocations look?

    1. Adrian Stone says:

      Methods and fields are actually exposed in a very similar way. Methods can be serialized with a generic serializer interface that receives callbacks to read every parameter to the method and then to write every return value from the method. Methods can also be cast to concrete types explicitly templatized on the method’s signature. This is done using the reflection libraries own safe cast mechanism similar to dynamic_cast.

      Built on top of these basic capabilities we provide helpers to serialize methods to/from strings (for very simple console-type invocation), to bind methods to boost::function objects, and to bind methods to script functions. In the case of script we use Lua, but any such language could be supported without extending the reflection library itself.

  7. NA says:

    What do you mean with “Reflection metadata had to be available at static initialization time”?
    Does this mean the meta data will be build outside of main()? When this is the case, you know you should’nt do big initializations oudside main?!

    You need also a general approach to set the values, how do you have solved this?
    How do you access this data from outside? e.g. When the GUI set the data. How do you do this with enums?

    1. Adrian Stone says:

      Yes, the reflection definitions for classes is generally constructed during static initialization time. I agree with you in general, that work outside of main() should be avoided, but for low-level systems like memory managers and dynamic_cast replacements it is more dangerous to rely on clients of the code to never use it during static initialization. Instead care must be taken to ensure the code behaves correctly no matter how it is used.

      Regarding setting data, there are a number of generic mechanisms for doing so. All fields and methods can be accessed through generic “serializers” like in built-in reflection systems in other languages. We have implementations of serializers that set fields or call methods from string data, XML data, custom binary file formats, etc. In addition to that, fields and methods themselves can be queried as objects and dynamically cast using the reflection system to create typed variable or function objects. So a field of type int can be safely acquired as a Variable object, and then can be worked on through typed setters and getters: “myIntVariable.Set(-4);” or “int x = myIntVariable.Get();”

      Enums are fully supported. Each enum has its own reflection definition, so every enumeration value can have unique properties. The serializer can then serialize enumerations as strings for optimal compatibility or as values for optimal efficiency. The same is true for the typed Variable objects I described above.

  8. NA says:

    Would be nice to see an article where you can describe the internals of the system, so the really interesting stuff.

    But I have again some questions:
    Where do you use meta template programming? I mean for this operator overloading:
    [
    Prop::Save() + Prop::Copy()
    ]
    you will definitely don’t need it.

    What approach to you use to bind the values? It is storing the offset address and the type of the variable?

    What I don’t get is, how can you invoke any arbitrary method?You have N number of arguments and Z argument types..
    You use some kind of variant data type for it, don’t you? I don’t see any proper solution without it.

    I would also like to email this, but I cant find your address. 😀

    1. Adrian Stone says:

      The system is pretty flexible in how it can bind variables. The most efficient option available is a raw data pointer with offset from the containing object. It can also, however, use pointer to member variables or even functors to expose data requiring custom accessors as variables.

      There are two mechanisms for invoking arbitrary methods. Methods can be dynamically cast to boost::function objects if relatively quick, type-safe access is needed. They can also be called generically with an abstract “stack” interface from which the method can request variables of different types. This is similar to how C# reflection works and how C functions are exposed to Lua.

  9. NA says:

    A dynamic_cast to a boost::function?
    never known that this is possible.

    with the stack interface you know something like:
    MethodBase.Invoke (Object, Object[]) ?

    But therefore you have to use something like a variant?

    Whats the common base type for the boost::function and for the “stack” call?

    1. Adrian Stone says:

      Reflected objects have a base class Reflection::Object. Reflected fields have a common base class Reflection::Field and similarly reflected methods have a common base class Reflection::Method. A list of Field and Method pointers can be accessed from an object’s reflection definition, and they provide abstract access to fields and methods of concrete types. Since Field and Method are themselves reflection objects, they can be dynamically cast to fields and methods of concrete types (for example, TypedField), and from there typed accessors like boost::functions can be requested.

      The abstract Method and Field objects have virtual functions for invocation that take an abstract ISerializer interface. So invocation might look like Method::Invoke(Object*, ISerializer*); The ISerializer interface has methods for reading and writing values. It could be described as a variant I guess.

  10. NA says:

    Did you get my message?
    I also thought about the use case for reflecting private members.
    When you have a setter/getter for a Member did you reflect the method and the field?
    How can you distinguish the use in the GUI or scripting, or is reflection of private members only useable for serialization?

    1. Adrian Stone says:

      Fields can be implemented in a variety of ways (raw pointers to members or accessors), but they always appear to users of the reflection object the same. However, since fields can be exposed multiple times with unique names and properties, they can be exposed differently to different systems. For example, a single class member m_someInt, could be exposed to file I/O as a pointer to member and to UI as a pair of blocking setters/getters. It could even be exposed to scripting twice, once using the pointer to member representation under the name “someIntUnchecked” and again with setters/getters under the name “someInt”.

      The notion of public/private access within the reflection definition is different from the notion in C++ because reflected elements simply carry an arbitrary set of properties that are interpreted in whatever way is appropriate for the system using them.

Reply in Thread