Building a Reliable Voice Transcription Pipeline for Indian Courtrooms (Part 2)

This is the final part of a two-part series on how we built a reliable voice transcription pipeline that works over unreliable, spotty internet connections. In case you haven’t read the last one, here is the link.

Let us quickly recap the problem statement once more:

1. We want to have a bidirectional stream that sends audio chunks from the browser to the server and receives transcript chunks from the server to the browser.

2. We want this system to work without data loss over unreliable, spotty internet connections.

In part 1, we went over the intricacies of the WebSocket initialization and how to set up and implement keep-alive correctly.

In this part, we will focus on the protocol that is used to keep track of audio and text chunks, and also the high-level backend architecture.

Decoupling the transport from application state

The idea isn’t anything new, and has been pioneered by ZeroMQ quite a long time ago. We need to create an architecture that supports resuming lost packets whenever the system detects a connection drop. Of course, the user should be unaware of all this. We cannot expect a judge to stop dictating just because there was a WebSocket reconnect.

We start by creating a concept of a requestID which persists across reconnects. The client retries with the same requestID whenever there is a reconnect as long as the page itself isn’t refreshed. We deliberately did not want to persist state beyond a refresh to allow the user an easy way to “wipe the slate” clean, so to speak. This requestID is then tracked on the server side to detect reconnect requests.

Resending lost packets

Next, we introduce the concept of an active queue and dead queue.

The active queue is nothing but a buffer between the Recorder and the Websocket. As mentioned earlier, the application cannot stop the user from recording whenever there is a reconnect. Placing an intermediate queue is the first step to let the audio chunks accumulate even if nothing is getting sent to the server. Of course, this has a limit, and if the internet hasn’t come back after a long time, the recording has to be stopped. This is what dictates the size of the active_queue. The larger it is, the more it lets us revive from longer internet outages, at the cost of higher memory/storage.

The dead queue is where the packets are pushed to after writing them to the websocket. This is nothing but a user-space TCP socket buffer. It is implemented as a fixed-size array which just stores the last n messages. How big the dead_queue will be is an important tradeoff. A larger dead_queue will definitely help when there’s severe packet loss, at the cost of higher memory/storage. Tune the size accordingly to your network statistics.

Now, with these two queues in place, we need a sequence_number to keep track of the packets that the client has sent to the server. This is implemented as a simple counter that increments for every chunk sent.

Mirroring the same on the server

Remember that the stream is bidirectional - we send audio and get back transcripts. So we have to mirror the same logic for transcripts on the server side as well. Interestingly, the same buffering ideas apply on the server side as well. That’s because the server will have to send the audio packets to an ML endpoint. And that endpoint will have fluctuations in how fast it can consume those packets. So we mirror the exact same concepts of active_queue, dead_queue, and a server-side sequence_number for transcript chunks. The queues are just Go buffered channels, and the dead queue is implemented as a ring buffer.

Combining all of it, this is how it looks

Note the amber colored sequence_number . That is for tracking audio chunks. And the green sequence_number tracks the transcript chunks.

The reconnect protocol

Let’s go over the protocol which is used when the system detects a broken connection.

Every connection has a requestID which is carried over across reconnects. For a fresh connection being created for the first time, this is how it looks:

1. We initialize a websocket connection like this: adalat.ai?requestID=abc123&seq_num=0

2. Then the server sends back READY 0 srv-42

3. And then normal operations of sending audio chunks and receiving transcript chunks begin.

The seq_num query param indicates the last received transcript sequence_number from the server. This is the green colored number above.

The 0 in the READY message is the last received audio sequence_number from the client. This is the amber colored number above.

The srv-42 is a random string to identify the server that sent the message. This is necessary in case the server crashes. Without this, it is not possible to distinguish between the server sending back the wrong sequence number due to an actual packet loss, or just because the server crashed.

Now, let’s say some amount of transcription has happened, and the client initiates a reconnect. This is how that will look:

1. Client → adalat.ai?requestID=abc123&seq_num=10

2. ← Server READY 3 srv-42

3. The server will compare 10 with its own counter. Let’s say the internal counter is at 12. Then the client hasn’t received 2 packets which it has already sent.

4. ← Server (2 missing transcript chunks)

5. The client will compare 3 with its own counter. Let’s say the internal counter is at 5.

6. Client → (2 missing audio chunks)

7. Then normal operations begin again.

If the server ID differs from what it has already stored, it forces a fresh connection and sends an appropriate error message to the user.

The reattach on the server side

The last piece of this puzzle is how to safely decouple the server-side state from the websocket connection and reattach it to a new connection.

As mentioned in the diagram above, everything inside the blue dotted square is what we call the orchestrator. It has 2 buffered channels and a ring buffer which stores all the necessary state for a given connection. Then we have a map of the requestID to this orchestrator struct.

The simple scenario is when the connection breaks, we park this orchestrator struct separately. And then when a new connection with the same requestID is seen again, we reattach the same orchestrator, and it simply resumes.

But the client and server are totally separate. A client-side disconnection does not necessarily mean that the server connection has also disconnected. So there are cases where, even without the server-side connection getting disconnected, a new request comes with the same requestID again. When that happens, we do what we call a live takeover.

We have to close the existing connection because it’s dead anyway. And then move the orchestrator to the new connection. This is harder than it sounds because you have to ensure the orchestrator state isn't polluted by consuming data from the stale connection or sending data to the old connection. You need to drain all connections safely, and you must reset any existing state in the current orchestrator to handle the new connection.

Moving to a checkpointing protocol

Keeping buffers on the server and client has one big flaw: there’s no right buffer size. The tradeoff is between memory and reliability.

An improvement we plan is to add regular checkpoints between the server and the client, so both sides know exactly how much was received on the other end. Then when a disconnect happens, we just have to resend from the last checkpoint. This ensures that reconnect will always be lossless. Right now, as implemented, if the number of lost packets exceeds the buffer capacity, the data is lost forever. This mechanism avoids that.

Conclusion

This brings us to the end of the 2-part series. The whole idea sounds really simple, and it really is. But the real challenge is implementing it correctly for the real world. Hopefully, these posts gave you some takeaways that we have learned the hard way.