Todd Schiller

Software, Knowledge, and Data Engineering


Library use is common but documentation is scarce. Your ability to ascertain information quickly about a library can mean the difference between success and failure in time-sensitive projects.

There are two cases where interaction with developers is the bottleneck: (1) reporting a bug, and (2) learning to use a feature. These are actually two sides of the same coin — in both cases, the role of the developer is to reconcile the software's intended behavior with its actual behavior.

Developers are busy. Communication with developers is asynchronous. There are often long delays between responses. Therefore, when contacting developers you should aim to minimize (1) the number of developer responses required, and (2) the effort required for a developer to respond.

The Developer Interaction Checklist

Here is my checklist for effective development questions and bug reports:

  1. A (high-level) overview of the problem you are trying to solve
  2. How you tried to solve the problem
  3. The exact error you received (and the inputs that resulted in the error, when possible)
  4. Your current understanding of the error, and if possible, how you've already tried to resolve the error (and what you've learned)

These steps are effective because they validate and verify the software:

  • Steps #1 and #2 validate: is the software supposed to be able to do what I am trying to do? If not, is there another (or better) way that I can accomplish the goal?
  • Steps #3 and #4 verify: how / why isn't the software doing what it's supposed to do?

In practice, you can often solve a problem yourself by following these steps. If you can't, you've put the developer in the best position to quickly address the problem with their unique knowledge.


Here's a question that a colleague and I asked on the DynamoRIO mailing list. Can you spot each of the four steps?

Is it possible to free memory in the application from a DynamoRIO client that wraps the free method? We are trying to defer all frees in the application until another method is called (pthread_mutex_unlock, in our case), or the thread exits. We're aware that this breaks transparency, but we still need to do it.

We're currently trying the following:

  • Wrap the pre-state of the free(...) method to record the memory address in TLS, and then skip the call to the wrapped function
  • Wrap the post-state of the pthread_mutex_unlock to perform the deferred frees on the recorded memory addresses by calling free(...)

The free(...) call in the client causes the following error in debug mode:

<rank order violation dynamo_areas(readwrite)@/src/core/vmareas.c:1544 acquired after global_alloc_lock(recursive)@/src> /core/heap.c:307 in tid:4ce0>
<Application test (19669). Internal Error Internal DynamoRIO Error: /src/core/utils.c:607 (dcontext->thread_owned_locks->last_lock->rank < lock->rank IF_CLIENT_INTERFACE(|| first_client || both_client)) && "rank order violation" ... >

, suggesting that the call to free(...) is being passed through an internal DynamoRIO method which is not re-entrant.

Note how in the email, we're not sure whether or not the behavior we're seeing is a bug. This is because we're not sure how the software is supposed to behave!

Based on this email, the developer was able to quickly get us unstuck (the same day!) using their unique knowledge:

Most likely your client call to free is a static import and is thus redirected to DR's heap allocator. A workaround is to call free in some way other than via a static import. Note that technically there are deadlock risks with acquiring application locks while in a DR or client context.

Xref two relevant feature requests:


You can make your interactions with developers more effective by including the information that best clarifies: (1) what the software is supposed to do, and (2) why / how / when the software doesn't do what it's supposed to.

This post was inspired by my friend Adrian Sampson's Grading Rubric for Bug Reports.