This is a continuation to my previous post on socket_wrapper project. There I mentioned about some upcoming improvements/feature additions to the project. Yes, it has been a while but we have achieved a great milestone. With some modifications to the initial design we now have basic support for fd-passing which was crucial for Samba project to test SMB3 Multichannel feature in its selftest environment. This article briefly tries to explain the approach used to implement fd-passing support inside socket_wrapper. I would suggest readers to go through my initial article on socket_wrapper project itself so that next few paragraphs make more sense.
Initial design
Since sole purpose of getting fd-passing support in socket_wrapper was to be able to send and therefore share socket file descriptors, it is a basic requirement to have all required metadata available to different(unrelated) processes in a machine. Thus comes the need for a shared store. Having such a centralized location will surely help in achieving our intention to share socket file descriptors among different processes. It is perhaps the most integral part in implementation steps. But at the same time concurrent access is also an issue. Thus synchronization using pthread robust mutexes is a must. We started off with these in mind to send sockets via sendmsg() and recvmsg() wrappers in socket_wrapper. Sender would then send indexes to our shared store where necessary data is stored. Receiver then tries to fetch based on received ancillary data. Of course all those actions has to be synchronized with the help of mutexes around shared store access patterns.
As we moved on
Over time it came to our attention that things can be much more simplified at least for Samba use case where socket file descriptors might not be consumed at same time. SMB3 Multichannel implementation in Samba follows this route so that a particular socket fd if gets passed on to another smbd(Samba daemon) process it is guaranteed to be used only by the receiving end. Thus we may not have to dig into shared store synchronization techniques and complex pthread mutex logic. At the same time if we manage to get rid of a shared store we have to recreate socket info structure at the receiving end. This way we reduce complexities with shared memory mapping and mutex synchronization.
Current implementation
sendmsg() wrapper inside socket_wrapper prepares ancillary payload with socket file descriptors that are to be passed on. Additionally it creates a pipe thereby placing its reading end as the final file descriptor in the cmsg ancillary data array. Now it writes the corresponding socket info structure in its entirety using the write end of pipe created earlier and closes it. recvmsg() wrapper on the other side extracts the file descriptor array and reads complete socket info structure contents from the pipe using last file descriptor received(which is the read end of pipe). Thus we recreate complete socket info for those passed socket file descriptors in memory for receiving side. Note that we are not having any pthread robust shared mutex protection as these sockets are presumed to be consumed only by one process at a time.
socket_wrapper version 1.3.0 was tagged with fd-passing support just to discover that we had memory leak and race condition issues while attempting to run current Samba selftest. Thanks to Stefan Metzmacher from Samba team who did great job in adapting the design to suite Samba needs and fixing issues as soon as they popped up. With his help we now have an extensive test coverage within socket wrapper selftest for fd-passing implementation. I hope the next point release that is to be tagged(v1.3.1 or v1.3.2) will be ready enough to get SMB3 Multhichannel tests run within Samba selftest using socket_wrapper.
Openforums 