mr-edd.co.uk :: horsing around with the C++ programming language

A problem with boost::thread_specific_ptr

[17th February 2010]

This week I hit a problem when using boost::thread_specific_ptr. The short story is, don't use one unless it's in static storage.

I was juggling some algorithmic code about in an attempt to parallelize it. Essentially, there was a large loop, each iteration of which would modify some element of a std::vector:

std::vector<Data> output(input.size());

for (iter_t b = input.begin(), e = input.end(); b != e; ++b)
{
   // do something with *b and modify 
   // arbitrary elements of output
}

In order to split up the work so that it could be fed to a thread pool, I decided to have each thread write to its own copy of the output. Once all the threads had done their work, their efforts could be recombined back in to a single vector in a (much quicker) serial step.

Now the phrase have each thread write to its own copy of the output screamed boost::thread_specific_ptr at me. So I created a little wrapper that initialized each thread's local data to a copy of output:

template<typename T>
class thread_local
{
    public:
        thread_local(const T &initializer);

        T &get(); // returns a reference to the thread-local data

        // iterators that yield the different thread-local objects
        const_iterator begin() const;
        const_iterator end() const;

        // ...

    private:
        boost::thread_specific_ptr<T> ptr_;
};

I also created another little class that's rather similar to boost::reference_wrapper that provided references to the thread-specific data dealt from a thread_local object.

Anyway, in the end the parallel version looked a little something like this:

std::vector<Data> output(input.size());
thread_local<std::vector<Data> > tl_output(output);

parallel_for_each(
    thread_pool, input.begin(), input.end(),
    bind(&loop_body, _1, tlref(tl_output))
);

// serial pass to recombine the data
BOOST_FOREACH(const std::vector<Data> &data, tl_output)
{
    merge(data, output);
}

Although it's a little more long-winded than the original, I don't think it's a bad effort. It seemed to do the job quite nicely at first, but weird things started happening when executed more than once.

The problem, I eventually found, was that a boost::thread_specific_ptr uses its address as key. Each time through the code, the boost::thread_specific_ptr inside the thread_local object was given the same address on the stack and therefore the same key. In other words, I had two objects that thought they were the same.

This actually turns out to be extremely dangerous since a boost::thread_specific_ptr<double> and a boost::thread_specific_ptr<std::string>, for example, can occupy the same address at different points in a program. Thus, the latter object would think that the data associated with a given key is a std::string when in actual fact it's a double. It won't take long for all hell to break loose after a cast like that.

I started a discussion about this on the boost mailing list. It's not clear yet whether the problem is a documentation error or a design error (I would say the latter), but it bit me pretty hard so I thought it might be a good idea to share it. There's also an existing Trac ticket.

Comments

Kirit

[22/02/2010 at 15:25:58]

As a poor man's TLS I used to use a map from the thread ID, something like map< thread_id_type, data_type*>. It's not nearly so elegant, but does at least mean that auto versions behave predictably. There is of course a load of nasty read/write locking needed around this as well depending on how you set it up, but at least it would only appear in your wrapper, and wouldn't necessarily cause any undue performance problem when actually executing.

FWIW I think I would have to agree that disallowing non-static use is a design flaw.

Edd

[22/02/2010 at 19:34:28]

That is indeed exactly the workaround I'm employing for the time being! For its current uses it seems to be holding up just fine.

I'd rather have something tried and testing by a large community, though. Here's hoping it's amended soon...

(optional)
(optional)
(required, hint)

Links can be added like [this one -> http://www.mr-edd.co.uk], to my homepage.
Phrases and blocks of code can be enclosed in {{{triple braces}}}.
Any HTML markup will be escaped.