On Asynchronous, Non-Blocking I/O

Non — blocking, asynchronous I/O APIs make your applications more resilient, responsive and cost effective. In this blog post we will look at the problems blocking I/O is causing, at how non-blocking I/O solves the problems and then apply these principles to the every-day use case of file uploading.

Synchronous Blocking I/O

Traditional I/O operations are blocking. This means that while data is being transferred between main memory and peripheral devices such as network adapters or storage controllers the thread that has initiated the operation is blocked and the CPU is waiting idle for the I/O operation to complete. Under certain conditions, this behavior can trigger catastrophic side effects.

Suppose you serve web requests using blocking I/O: each request is associated to a thread. When a web request is accepted, a thread is acquired from a thread pool. The application will use that thread to run database queries, invoke downstream services and so on until all information required to put together a response become available. After a response is provided, the application will release the thread back to the thread pool in order for it to be reused.

This takes a fair amount of time, and often most of this time is being spent waiting for I/O operations to return. If you experience a spike in requests, threads will be taken out of the thread pool faster than your app can return them. As a result, you will exhaust your thread pool very quickly and new threads will be created to serve incoming requests.

Creating a new thread is expensive and will increase response latency even more. In addition to that, as the number of threads increases, context switching will take more and more time. Your system will spend more time trying to figure out which thread needs to be scheduled next and setting the stage for it than processing requests and will eventually become non-responsive.

Blocking I/O also costs you more money. While CPU cores are waiting idle, polling for the result of the blocking I/O operation, they cannot be used to perform useful calculations. As a result, you will need more CPU cores to run your business logic within service level objectives, which means you will spend more money than necessary.

Asynchronous, Non-Blocking I/O

A non-blocking I/O operation initiates the data transfer by sending instructions to specialized hardware components and return immediately. The hardware components will transfer the data in the background. When the transfer is complete results are returned by invoking a callback provided when the operation was initiated.

Because the I/O operation returns immediately the thread handling the request is not blocked, it can immediately be returned to the thread pool and reused. Another thread will be later used to run the callback delivering the results.

This solves the first blocking I/O problem we mentioned above: you will not exhaust the thread pool any more, your system will be more responsive and will handle request spikes a lot better and you will pack more compute cycles on the same number of cores because all that idle CPU time does not exist any more, making it cheaper to run.

With non-blocking I/O the CPU is not waiting for the I/O operation to complete. All those precious cycles wasted waiting for blocking I/O to complete can now be used to perform business logic. As a result you can run more business logic on the same number of CPU cores, and your system becomes more cost effective to run.

Real World Use Case: Non-Blocking HTTP File Upload

One of the use cases where most the IO / compute workload ratio is very high is uploading file to web server.

With blocking I/O the thread handling the connection will first wait for chunks of data to arrive over the internet, then wait again for each chunk to be written to disk before returning a result to the caller. When the result is ready it will wait for it to be delivered over the network before the request becomes complete and the thread returned to the pool.

With non-blocking I/O the flow looks very different. After the connection is accepted, chunks of data are being delivered to a callback as they arrive over the network. That callback will initiate a non-blocking write to the destination file, and return immediately, freeing the thread. When the write operation completes, another callback is invoked with the result of the operation. After the last write operation succeeds, the response is being written to the socket.

If you think the flow above sounds like callback hell, you’re right. Luckily frameworks such as RxJava or Project Reactor simplify the task by making it possible to write that flow in a fluent, readable style.

WebFlux is a framework for reactive web programming base on Project Reactor. When used in conjunction with Spring Boot and the AsynchronousFileChannel class from the Java NIO framerwork, it enables us to upload files in a reactive manner, using non-blocking I/O operations exclusively.

I have put together a fully reactive, non-blocking file upload over at GitHub. We’ll have a close look at how it works in another post, until then you can have a sneak peek at ReactiveUploadResource.java.