Binder Threading Model
Binder threads
In the HamKingServer class of the HamKing sample project [1], I annotated all the service methods with @BinderThread:
When an application is started by Android system, a Binder thread pool is allocated for the application to handle incoming Binder transactions. Note that a Binder thread is only used to serve incoming transactions on the server side of a Binder service. When an app initiates the transaction on the proxy side, the app usually fires off and wait for the transaction to finish on the UI thread. In the above example, the four annotated methods run when the client app calls the service methods remotely and they all run on Binder threads. This Binder thread pool frees main thread from caring about serving incoming Binder transactions so that it can focus on UI related works. However, for certain kinds of programs, if all it does is to listen and serve incoming transactions, it may not need a Binder thread pool at all. Because if the main thread spawns a thread pool to handle incoming calls, what should the main thread do? The Service Manager we talked about before is such a program where a Binder thread pool is not needed.
The Binder thread pool for an application is started early during process launching:
The AppRuntime represents an Android Runtime instance used by an application. The onZygoteInit is invoked during application launching. First of all, let's take a deep look into the ProcessState class. The IPCThreadState class we have seen in "Binder data model" is a class to manage the interactions with Binder driver on a per-thread basis. The ProcessState instead manages the global Binder state of a process.
The most important task done by a ProcessState during initialization is to open the Binder driver and use mmap to create kernel buffer mapping. The gProcess field is a global singleton of ProcessState, it only gets initialized the first time ProcessState::self is called. On line 42, open_driver is called to open the /dev/binder device. On line 29, an ioctl system call is used to check for Binder driver version. Another ioctl call on line 35 is used to set the maximum number of allowed Binder threads that driver can request user space to spawn. After driver is open, line 54 calls mmap to setup a kernel buffer mapping for current process. The size of the buffer is about 1 MB.
The startThreadPool method calls spawnPooledThread to start a Binder thread. The isMain indicates whether this thread joins thread pool voluntarily. In this case this is set to true for this reason. Binder driver can ask user space to spawn a new thread to join the thread pool when all threads in the pool are busy. isMain will be false for these kind of threads. The maximum number of threads that kernel can request is set by BINDER_SET_MAX_THREADS command in ioctl. PoolThread class inherits from the Thread class in Android system utils library. The way this Thread class works is that it will run code inside threadLoop continuously until threadLoop returns false. The PoolThread class returns a false so the threadLoop will only be called called once. But the joinThreadPool method on IPCThreadState will never return unless an error happens.
The joinThreadPool method first uses a BC_ENTER_LOOPER or BC_REGISTER_LOOPER command to register itself to Binder driver then run in an infinite loop to process incoming commands unless an error happens. If an error happen then the Binder thread just uses BC_EXIT_LOOPER to remove itself from the thread pool. We touched on talkWithDriver in the article "Binder data model", basically this method uses a BINDER_WRITE_READ command code with ioctl to write and read data to and from Binder driver. executeCommand then processes the data that are read from the driver. The BINDER_WRITE_READ command is blocking, which means it may result the calling Binder thread to sleep and wait if there is currently no work for the Binder thread to do (which means the thread can read no data from driver). So in short, the way a Binder thread works is to write, fetch and process data in an infinite loop and that's why the term LOOPER is used in the command name.
Now let’s look at the BC_* and BR_* command codes. Until now you have seen three levels of command codes. The top level is the command codes to identify the work type for an ioctl system call. For example, the BINDER_WRITE_READ, BINDER_SET_MAX_THREADS and BINDER_VERSION are several such codes we have seen so far. BINDER_WRITE_READ command for ioctl is a general command type to use when a thread wants to read and write data from and to the driver. So BINDER_WRITE_READ defines a second level of BC_* and BR_* commands to indicate the action type of this BINDER_WRITE_READ. ioctl knows nothing about these BC_* and BR_* commands. Probably the most important BC_* and BR_* command type is BC_TRANSACTION that is used when a process initiates a Binder transaction and BR_TRANSACTION that is used when a thread processes incoming transactions. A Binder transaction defines a third level of commands to indicate the exact action to perform in the transaction. For example, the IHamKingInterface Binder service in HamKing project implements three service methods: requestOrder, pickupOrder and cancelOrder. Each method corresponds to a transaction command code. BINDER_WRITE_READ knows nothing about these transaction command codes.
Now let’s look at the BC_* and BR_* command codes for BINDER_WRITE_READ.
The acronym BC stands for "Binder Command" which means the data goes from user space to Binder driver. The acronym BR stands for "Binder Return" which means the data goes from Binder to user space. Let's take a Binder transaction for example, when a process initiates a transaction, source process sends Binder driver a BC_TRANSACTION with a binder_transaction_data data structure as it's payload. Binder driver will put a BR_TRANSACTION command with payload data to the target process's todo list. A Binder thread in the target process will read this BR_TRANSACTION from driver and process the transaction in user space. After the processing is done, the Binder thread will use BC_REPLY to send a reply to driver together with reply data. The driver processes this BC_REPLY by locating the source process of this transaction and put a BR_REPLY into the source process's todo list. Once the source process receives the BR_REPLY it knows the transaction is finished. The source process can send a BC_FREE_BUFFER to request driver to free the kernel buffer used for the reply.
Threading in Binder driver
Going back to the Binder thread registration process. The BC_ENTER_LOOPER or BC_REGISTER_LOOPER command will be sent to kernel in the next talkWithDriver, let's take a look at how kernel handles this command:
Line 17 reads out the command code for this BINDER_WRITE_READ and adjusts the pointers on line 19 and 40. As we can see, the driver handles the two commands simply by setting a flag on the binder_thread data structure representing the current thread. The only difference is that for BC_REGISTER_LOOPER the driver keeps track of the number of threads it requested user space to start and the number of threads that are started in response to such requests. With that being said, a binder_thread structure will be created to keep track of each thread that interact with Binder driver even if the thread is not a Binder thread. The main thread of a process is usually used to fire off a Binder transaction but the main thread is not a Binder thread. For example, when you use startActivity to launch an Activity on UI thread, the UI thread has a corresponding binder_thread in Binder driver but the UI thread is not a Binder thread. However if a thread wants to read and process incoming data from Binder driver it must first register itself as a Binder thread, otherwise the driver won't consider the thread to schedule transaction works.
The binder_get_thread function gets the binder_thread data structure in current process or creates a new one if none exists. Let's first look at the declaration of binder_thread:
The proc field points to the binder_proc structure it belongs to and the rb_node field is used to link this binder_thread structure onto the node tree inside the binder_proc. The waiting_thread_node is used to link the structure onto the waiting_threads list inside the binder_proc. A Binder thread that is linked to this waiting list indicates that this thread is idle. The pid field is the Linux process identifier of this thread. The looper is a bitmap indicating the thread running states. looper_need_return indicates whether an ioctl should return to user space instead of waiting for work to do. For example, when a thread uses BC_ENTER_LOOPER or BC_REGISTER_LOOPER to register itself as a Binder thread, the calling thread isn't ready to block and wait for work. In this case, the ioctl returns immediately instead of putting current thread to sleep and wait for work. transaction_stack points to a stack of binder_transaction items that the current thread is processing. It is a stack because Binder transactions can be recursive: A thread can fire off another transaction while in the middle of processing an incoming transaction. The wait field is a Linux wait queue where the calling thread will sleep on when there is no work for the thread to do. The todo field is a list of binder_work items for this thread. The binder_proc also has such a todo list. While scheduling a work item, the driver tries to select a best thread to schedule it onto. If such a best thread cannot be found, the driver will just put the work item onto the binder_proc's todo list so that a Binder thread in that process can pickup the work item. Besides, if a thread is not a registered Binder thread no work shall be scheduled on to that thread's to do list, because such a thread will never read its todo list to process. Consider this code snippet in HamKing project:
You shouldn’t ever need to call linkToDeath on a remote service if it is acquired through bindService since you will get a callback in onServiceDisconnected anyway when remote process dies. However we will just add a death listener for demonstration. After you add an order, kill the server app process and you will see a line of log like:
E/MainActivity: linkToDeath called on Binder:25869_3
The UI thread of the client app registers a death listener to the Binder driver. On initial thought it makes sense for Binder driver to deliver the death notification to the UI thread’s todo, but the Binder driver cannot do that since the UI thread is not a Binder thread. Instead, the work item for the death notification is put on the todo list of the client app process. A Binder thread inside the client app later picks up the work and handles the death notification.
We have looked into the binder_thread data structure, let's now look inside binder_get_thread function:
The binder_get_thread function first tries to find a binder_thread data structure in current process that has a match pid. If such a binder_thread doesn't exist, then create a binder_thread structure and link it to the binder_proc's tree.
The Binder driver may request user space to spawn new threads to join the Binder thread pool:
Before binder_thread_read finishes, driver checks whether it should request user space to spawn a new thread to join the Binder thread pool. If so driver will put a BR_SPAWN_LOOPER command in current thread's read buffer to let user space process it. Line 8 checks whether there is no idle Binder thread left in current process. Line 9 checks whether the number of requested Binder thread has reached the limit set by user space. After the ioctl is returned, user space will handle it by spawning a Binder thread and join the thread pool:
We have seen the spawnPooledThread earlier in this article. Note that the isMain argument is set to false this time which means this Binder thread is created in response to a driver request. IPCThreadState will use BC_REGISTER_LOOPER command code to register the Binder thread instead.
It is important to understand that the action of writing data into the driver is not blocking. Which means the write action either succeeds or fails but will not put the calling thread to sleep. On the other hand, reading data from Binder driver is blocking. If there is no data for the current thread to do then the calling thread will be put to sleep. It is important that binder_ioctl_write_read processes data write earlier than read, otherwise the system may dead lock:
Since Android framework doesn’t set the O_NONBLOCK flag when opening /dev/binder, the non_block argument will always be false. binder_available_for_proc_work_ilocked checks whether current thread is available to handle work items on binder_proc's work list. If a thread is a registered Binder thread and not have work to do then this function returns true. binder_has_work_ilocked checks whether there is work to do for current thread. The binder_wait_for_work is where the current thread might be put to sleep. These steps are just typical steps to put a thread to sleep in Linux kernel. If there is no work for current thread to do then the current thread will be added to the wait queue in the binder_thread structure and the binder_proc's waiting thread list. When line 74 is called, a context switch will be called and the current thread will be put to sleep.
While an Android app is running, most of the Binder threads are just sleeping to wait for work to do. For example, this result is typical when you check the Binder threads state of an application:
The S in the "S" column means the thread is sleeping and the binder_thread_read means the thread is sleeping in binder_thread_read function.
When a work item is added to a target thread’s todo list, the driver will wake up the target thread and then the target thread can continue reading data into the read buffer. For example, binder_proc_transaction will be called during a transaction when Binder driver has found a target thread to perform the transaction:
The t parameter in binder_proc_transaction represents the current transaction to schedule. The proc parameter represents the target process to schedule on and thread parameter represents the target thread to schedule on. An empty thread parameter means there is no best thread to schedule the transaction on, so binder_select_thread_ilocked is called to select one thread to schedule the transaction on. The function just selects the first Binder thread that is idle. Line 13 to 16 adds this transaction item to the thread's todo list or the process's todo list. Then binder_wakeup_thread_ilocked will be called to wake up the selected thread. After the thread is waken up, it can continue to run inside binder_thread_read and pickup the transaction item in the thread's todo list or the process's todo list, write it the read buffer and return to user space.
Binder work items
During this article I keep using the phrase “work item”. Of course that term indicates some job for a thread or process to do. Let’s see how a work item is abstracted in Binder driver. The work item is represented by a binder_work data structure in Binder driver:
The binder_work structure is surprisingly simple. The entry field is used to enqueue this work item onto a binder_thread's todo list or a binder_proc's todo list. Besides that there is only a type field to represent the work type. For example, the BINDER_WORK_TRANSACTION means the work item is a Binder transaction and BINDER_WORK_DEAD_BINDER means the work item is a death notification. Since this structure is so simple, it cannot be the full story. The way this binder_work works is to be embedded in other data structures:
Note that the binder_work structure is just embedded in another data structure which mean the work fields are not pointer types. To view it in an object oriented way, this means the four data structures inherit from binder_work so they gain the capability of being scheduled onto a todo list. For example, during a transaction the driver will create a binder_transaction and link it the target todo list. When the target thread processes its todo list, it will check the type of the binder_work and "cast" the binder_work to the container binder_transaction. The actual code is simple:
Line 14 to 18 selects the target list and line 20 dequeue a binder_work from the list. Line 21 checks the type of the work item then line 23 "casts" the binder_work to binder_transaction.
Binder polling
The benefit of using Binder thread pool is to free the main thread from serving incoming transactions. However if the main business a program does is just serving incoming transaction, then it doesn’t make sense to create a thread pool. The Service Manager is an example of this kind of program. The single core functionality of Service Manager is to serve other process’s requests of adding and fetching system services. It is a single threaded program where all Binder transactions are served on the main thread.
Note that on line 41 the size of the thread pool is set to zero and the main thread is responsible for serving all incoming Binder transactions. A Binder thread sleeps inside binder_thread_read until a work item is enqueue onto its todo list. However Service Manager cannot act that way since there they other jobs the main thread needs to do, otherwise the main thread won't have a chance to do other jobs. Binder polling provides a mechanism for a process to receive a callback when there is something for it to read, instead of letting the thread sleep waiting for data. The Looper is Android's solution of event notification and it is widely used in various places like message queue and input system. It's implemented with the help of the Linux epoll system. Don't confuse it with the term LOOPER used in BC_* commands though. Looper deserves its own article so we won't deep dive into it here. Basically you can add some file descriptors to a Looper and receive a callback when something on the file descriptor happens. For example, line 10 to 14 says that when /dev/binder has some data for me to read, invoke the callback cb.
Under the hood, Looper uses epoll to poll on the state of /dev/binder and the exact behavior is implement by Binder driver:
After the pollAll method is called on line 49, Looper will call into the kernel with epoll and hit binder_poll. Line 16 checks whether there are work for current thread to read. If there are, then EPOLLIN will be returned and eventually it causes the handleEvent function in BinderCallback to be invoked. Otherwise 0 is returned indicating the device is not readable for calling thread and the calling thread may go to sleep. Yes, Looper will still put the current thread to sleep, but the key is that you can observe multiple event sources not just the /dev/binder readable event from the main thread. Line 14 adds the binder_thread's wait queue to a poll table so that when there is work item for this thread to do the thread can be waken up and poll the Binder device again.
Let’s take a look at the setupTo method in BinderCallback. The setupPolling method is called to prepare for polling on /dev/binder:
Basically it prepares polling by registering the main thread to Binder driver. Otherwise the driver will never enqueue work the binder_thread data structure of the main thread and the thread will never be waken up when polling. Even though the main thread also uses BC_ENTER_LOOPER to register itself like the threads in Binder thread pool, how they interact with Binder driver is very different. At last, let's look at the handleEvent method in BinderCallback:
We have seen the getAndExecuteCommand method before. This method just reads data from Binder driver and process it until all data in the input buffer is processed.
