Migrating existing C++ code to use modules

Microsoft released experimental support for modules a while ago. But it was not until the recent launch of Visual Studio 2017 that it provided us the modules that implement the standard library which allows us to try out modules much easier. They would only be installed if the component “Standard Library Modules” is explicitly checked during installation.

There are several posts about modules in the Visual C++ blog, but all of them are more of a hello world than anything else. In this blog post I would like to provide a bigger example, closer to the real world. We would migrate code implemented without modules to C++ with modules. The code simply parses this configuration file:

<configuration>
  <server address="127.0.0.1" port="8080"/>
</configuration>

that would allow to configure the connection details of a TCP client. It will use tinyxml2 to perform the XML parsing.

This was the design of the library (without modules):

clientconfig_legacy

I posted all the code in github. The legacy code (without modules) will not be discussed. This will be design of the library using modules:

clientconfig

Our goal is to create a static library, clientconfig.lib that will be consumed by its users using clientconfig.ifc (a module interface file, following the Internal Program Representation binary format). Note that for users to consume the library, several variants of the .ifc file may be created, i.e. depending on the configuration (Debug or Release) or the platform (Win32 or x64) we could have for instance 4 variants. In this case the user of the library will be a simple program, client.exe, that will only read the configuration.

clientconfig.lib will be implemented using a private module, tinyxml2. clientconfig.cpp will consume that module through the tinyxml2.ifc file and it will export the declarations of the whole library in clientconfig.ifc. Because exports are not transitive (unless explicitly requested with an “export import” statement), tinyxml2 symbols will not be seen outside the clientconfig module.

Before we start delving into the code, some important advice (that applies as of VS 2017 update 2). There are some issues if modules are used from the IDE, so do not use the IDE at all when testing modules. Also use Release mode or you would not be able to link executables. All the commands that will be used below are supposed to be executed in a VS2017 native x64 console. The github repo contains nmake makefiles with the commands as a reference.

The first step was to port tinyxml2 to use modules. tinyxml2 5.0.1, which is the version I started from, can be cloned from here and I checked out the tag 5.0.1. The things I changed:

  • Starting from tinyxml2.h, renaming it to tinyxml2_desc.ixx and specifying the symbols to export; we will export the whole tinyxml2 namespace:

module tinyxml2;

export namespace tinyxml2
{

// ... declarations

}

  • Replacing #includes of C++ headers for import. E.g. cctype, cstdlib, cstring and were simply replaced by import std.core. Note that some includes, like stdio.h stay, this is because even if they are in std.core,
    they do not define some macros needed to be able to use the C Standard Library (e.g. SEEK_SET, INT_MAX…)
  • Getting rid of macros. Modules do not export macros. The TIXMLASSERT macro defined originally in tinyxml2.h is not exported by its counterpart tinyxml2.ifc so I replaced all of these macro invocations in tinyxml2.cpp with if ( !((void)0,(x))) { __debugbreak(); }. tinyxml2_desc.ixx does not need an include guard either.
  • Removing the omision of values for arguments with a default value. EDIT: I considered this a Visual Studio bug. But the according to the C++ modules TS issues list, this seems the way functions with default arguments are exported.
  • Making protected and private sections public, because otherwise trying to access them causes an undefined symbol error. EDIT: I considered this a bug at the beginning. But after watching C++ Modules: The State of The Union it seems intentional.
  • Some changes unrelated with modules: adding #define _CRT_NO_VA_START_VALIDATION to tinyxml2.cpp and also:

#define TIXML_VSNPRINTF _vsnprintf
#define TIXML_SNPRINTF _snprintf

as 2017 comes with an implementation for vsnprintf and it is no longer necessary to define our own.

Please follow the makefile to see how to compile the library with the compiler switches needed to support modules.
It is worth noting after generating an .ifc file, for instance with the command:

cl /c /EHsc /experimental:module /MD /std:c++latest tinyxml2_desc.ixx

Not only an .ifc object is created but also and .obj file containing symbol definitions needed to create the library.

The main interface description file of the clientconfig library is as simple as:

module clientconfig;
import std.core;

export void readConfigFile(std::string& address, std::string& port);

And the code to implement it:


import clientconfig;
import tinyxml2;
import std.core;
using namespace std;
void readConfigFile(string& address, string& port)

{
  tinyxml2::XMLDocument doc; XMLElement* configuration;
  XMLElement* node;

  doc.LoadFile("config.xml");
  configuration = doc.FirstChildElement("configuration");
  node = configuration->FirstChildElement("server"); address = node->Attribute("address", 0);
  port = node->Attribute("port", 0);
}

Error checking code was ommited as is not very relevant to the discussion. Also, this code is subject to change. The gcc implementation of modules suggests that the technical specification may be changed, so a file could be marked as the implementation translation unit of a module programmatically, but for now this seems to work on Visual C++.

The main.cpp will consume the clientconfig module:


import std.core;
import clientconfig;

using namespace std;

int main()

{
  std::string address, port;
  readConfigFile(address, port);

  cout << "Address: " << address << "\n"; cout << "Port: " << port;

  return 0;
}

This completes our program. I will be excited to see how the Microsoft implementation of modules scales when I apply it to large projects, as the build throughput should improve dramatically.

pct 1.0 now supports Visual Studio project files and is multithreaded

The initial version of pct (0.1.0) still required a lot of configuration to auto-generate precompiled headers for large codebases and it was not as fast as some users would have liked (#4) . Version 1.0 addressed both shortcomings.

pct can now parse Visual Studio project files to extract information like the path of the include directories to search for headers or the macro values to use; so the new options –sln or –vcxproj save you from specifying a long command line . Users that use qmake or cmake just need to generate Visual Studio project files from their build files to able to use the tool in the same way (See Cross-platform development with C++).

Performance greatly improved after, by processing every .vcxproj in parallel with std::async(). It is not uncommon to have big codebases in C++, so this change was useful and easy to implement.

I recently realized that environment variables referenced in the Visual Studio project files were not being expanded. Qmake expanded them for me before generating the Visual Studio project files, so I did not realize this until now; but it was  fixed on 8d43ad1. I also added an option –excluderegexp which is useful for qmake users, because it could be used to ignore moc_* files (which do not need to be parsed by pct because all their headers are referenced already in the original moc’ed header). E.g. –excluderegexp “moc_.*”.