Race condition Scenario 1

Recently we hit an issue where the client program has stuck. The client has been written in such way it sends requests to server in synchronous manner (sends the next request only after receiving the acknowledgement for current request).  This issue happens intermittently.  This symptom tells that there is some race condition.

On the server side, there are two threads

thread 1

  1. submits a request to one of its internal submission queue.
  2. increments the io_submitted value.

thread 2

  1. picks up the item from submission queue and does an asynchronous IO using libaio.
    1. libaio thread calls the call-back function passed to it once the IO is done.
    2. As part of call-back function aio thread enqueues the request into completion queue
  2. picks up the item from completion queue and increments io_completed value.
  3. the does a check io_submitted == io_completed to do next set of task.
  4.  after completing the next set of tasks, sends a response to the client.

The problem is that the client is not receiving the acknowledgment.  Why?

There is a race:   Before thread 1 increments the io_submitted value,  thread2 increments io_completed and does a comparison check. This can be possible if thread1 is scheduled out before we increment io_submitted value.

Couple of solutions:

  1. Move increment before submitting request o internal submission queue.
  2. Use spin lock to protect the io_submitted



Working with Docker

Docker is a orchestration layer on top of Linux containers. It creates a lightweight work environment like BSD jails, Solaris zones. You need to install set of packages in your ubuntu OS:


1. Create docker images from docker file
docker build -t image1 . ( Docker file is located at ./)

2. List out docker images

docker images

3. changes to images and committing changes

docker run -i -t –name guest image1:latest /bin/bash
apt-get install vim
docker stop guest
docker commit -m “message” -a “author” <container-id of guest> image1:latest

docker rm guest

docker run -i -t –name guest image1:latest /bin/bash

check if installed vim is installed.

dpkg –get-selections |grep vim
3. Update existing image
docker pull mysql
docker stop my-mysql-container
docker rm my-mysql-container
docker run –name=my-mysql-container –restart=always \
-e MYSQL_ROOT_PASSWORD=mypwd -v /my/data/dir:/var/lib/mysql -d mysql

your data should have been stored on -v volumes.

4. List containers

docker ps -a

5. Stop container

docker stop guest

6. start container

docker start guest

7. Aattach a running container, you exited from the session, but container still running

docker attach guest

8. Remove a container

docker rm guest

9. Reomve docker images

docker rmi $(docker images -f “dangling=true” -q)


In general,  egnieers use a script to start a container and share the directories from the host machine. Ex:-



mkdir -p $HOME/$1-shared

docker run –rm -t -i –name $1 -v $HOME/.gitconfig:/root/.gitconfig -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix:rw –privileged –net=host -v $HOME/.Xauthority:/root/.Xauthority -v $HOME/$1-shared:/root/shared  image1:latest /bin/bash

option description:

–rm  : remove container when it exit.

–name : name of the container

-i : keep STDIN even container is not attached.

-t : allocate a psudo TTY.

-v :  directory mapping from host to container (sharing of host directories)

Git Commands (Every day use)

Here are the list of git commands that I use every day:

  • Clone remote repository
  • list the branches in test-repo repository.
    • git branch -vv
  • Switch between branches
    • git checkout milestone1
    • git checkout milestone2
  • Create a new branch work1 from milestone1
    • git checkout milestone1
    • git branch work1
    • git checkout work1
    • You can combine above two commands with “git checkout -b work1”
  • Push new branch work1 to repository test-repo
    • git push origin
  • Merging branches (merge milestone2 code to work1)
    • git checkout milestone2
    • git pull (git fetch & git merge origin/milestone2)
    • git checkout work1
    • git pull
    • git merge –no-ff milestone2
    • git push origin work1
  • push a modification to file1 to work1 branch
    • git chekcout work1
    • git pull
    • git add file1
    • git commit -m “message” (or git commit -a -m “message” to skip staging area)
    • git push origin work1
  • Delete local branch
    • git branch -d work1 (to delete tracking branch)
    • git push origin  :work1 (to delete branch on remote repository)
  • You have done some changes in your local tracking branch, but you want to work on different branch and come back. If you have new file, then add it to the index.
    • git add newfile
    • Do changes in current indexed file file1.
    • git stash
    • git checkout master
    • do some work on master branch
    • git checkout work1
    • git stash pop (will pop both file1 and newfiles changes on top of work1 branch)
  • See the history of commits
    • git log
    • git log -p -1
    • git log –stat
  • Code differences
    • git diff origin/master (local uncommitted changes with remote tracking branch)
    • git diff master (local uncommitted changes with tracking branch, where you commits your changes)
    • git diff COMMIT^ COMMIT (compare commit with ancestor commit of tracking branch)
    • git diff –staged (staged changes with tracking branch recent commit)
    • git diff (not committed with staged)
    • git diff HEAD (upstaged changes with recent commit)
    • git show (to see the commited changes, difference with parent commit)
  • Status of your changes
    • git status
  • Resetting local committed changes
    • git reset <commit-id> (HEAD is moved to the specified commit-id)
  • Resetting local staged changes
    • git rm –cached
  • Discard changes in current working directory. (not staged)
    • git checkout — file1
  • Bringing changes to exiting commit
    • git add file2
    • git commit –amend (will add file2 change into same commit, not pushed)
  • Generate patch from commit
    • git format-patch -n1
    • git am <p

We need to understand some theory behind how git works. Some common terms we come across while working with git:

Repository, Remote tracking branch,  Tracking branch, Local repository

Read this nice blog at  tracking branches

Git index (aka staging area/cache/directory cache/staged files)

Read more about index here Git index


Good string programming puzzle

Problem Definition:

You are challenged to write an algorithm to check if a given string, s, can be formed from two other strings, part1 and part2.

The restriction is that the characters in part1 and part2 are in the same order as in s.

Test Cases:

  1. s = “Bananas from Bahamas” ;  part1 =”Bahas” ; part2 =” Bananas from am”
  2. s =”Can we merge it? Yes, we can!”; part1=” an mrgei? e,we”; part2=”C wee tYs can!”
  3. s=”Making progress; part1=”Mak pross”; part2=”inggre”
  4. s=’codewars’; part1=’code’; part2=’wars’
  5. s=’codewars’; part1=’cdw’; part2=’oears’



  1. Start matching character by character, there are two possibilities to start
    1. ‘s’ with ‘part1’
    2. ‘s’ with ‘part2’
  2. ‘s’ with part1:  Start matching string ‘s’ in part1 as long it matches and then move to part2 as long as it matches there. come back to part1 and try to match as much it matches then move to part2 … till the end of string ‘s’. If we can’t make progress then return False.
  3. ‘s’ with part2 :  If step2 fails, then start matching ‘s’ with part2…
  4. It is not just two possibilities, for every character match there are two possibilities, so we need to handle all those cases.
  5. For example:
    1. s = “Bananas from Bahamas” ;  part1 =”Bahas” ; part2 =” Bananas from am”
  6. For first character ‘B’ from s,  two possibilities s[0] == p1[0], s[0] == p2[0].  ‘s’ matches with ‘p1’  till “Ba”. The  rest of ‘s’ (“nanas from Bahamas”) does not match with p2 (“Bananas from am”) . so we have try with second possibility, s[0] == p2[0], this matches till “Bananas from ” in p2.  The rest of ‘s’ (“Bahamas”)  does not match with p2 (“am”). Then try to match “Bahamas” with p1 (“Bahas”), p2(“am”).
  7.    s=”Bahamas”, p1=”Bahas”, p2=”am”.  comparisons should go like this:
          'B' in s and 'B' in p1 (1)
               'a' in s and 'a' in p1   (2)
                  'h' in s and 'h' in p1  (3)
                      'a' in s and 'a' in p1 (False) 
                         'm' in s and 's' in p1 (False) 
                         'm' in s and 'a' in p2 (False) 
                      'a' in s and 'a' in p2 (True)  (4)
                         'm' in s and 'a' in p1 (False)
                         'm' in s and 'm' in p2  
                            'a' in s and 'a' in p1
                               's' in s and 's' in p1
                                  '' in s and '' in p1 (True) (5)

True value is bubbles up to (1) . In the order of 5,4,3,2,1

Python code:

Recursive solution:

def is_merge(s, part1, part2):
  if not part1:
     return s == part2
  if not part2:
     return s == part1
  if not s:
    return part1 + part2 == ''
  if s[0] == part1[0] and is_merge(s[1:], part1[1:], part2):
    return True
  if s[0] == part2[0] and is_merge(s[1:], part1, part2[1:]):
    return True
  return False

Iterative Solution:

def is_merge(s, part1, part2):
    queue = [(s,part1,part2)]
    while queue:
        str, p1, p2 = queue.pop() 
        if str:
           if p1 and str[0] == p1[0]:
              queue.append((str[1:], p1[1:], p2))
           if p2 and str[0] == p2[0]:
              queue.append((str[1:], p1, p2[1:]))
           if not p1 and not p2:
              return True
    return False

Aplication Architectures using Google cloud

The below architecture diagrams helps in understanding deployment architectures for different use cases.  I want to have a quick pointer to look at these diagrams from my blog, instead of searching for them every time.

Web applications:  https://cloud.google.com/solutions/architecture/webapp

Digital asset management and sharing:  https://cloud.google.com/solutions/architecture/digitalassets

Content Management : https://cloud.google.com/solutions/architecture/contentmanagement

High Performance Computing:  https://cloud.google.com/solutions/architecture/highperformancecomputing

IOT: https://cloud.google.com/solutions/architecture/streamprocessing

Mobile Apps and Games:  https://cloud.google.com/solutions/architecture/mobileandgames

CAP theorem

CAP theorem states that it is impossible for a distributed system to simultaneously provide all three guarantees of

  • Consistency: All nodes see the same data at the same time.
  • Availability: a guarantee that every request receives a response about whether it was successful or failed.

Availability in CAP is defined as “every request received by a non-failing [database] node in the system must result in a [non-error] response”

  • Partition tolerance: the system continues to operate despite arbitrary message loss or failure of part of the system.

A system is partition tolerant if processing can continue in both partitions in the case of a network failure


In above article, Robert Greiner, neatly explained the CAP with diagrams.

“Given that networks aren’t completely reliable, you must tolerate partitions in a distributed system, period. Fortunately, though, you get to choose what to do when a partition does occur. According to the CAP theorem, this means we are left with two options: Consistency and Availability.”

CP – Consistency/Partition Tolerance – Wait for a response from the partitioned node which could result in a timeout error. The system can also choose to return an error, depending on the scenario you desire. Choose Consistency over Availability when your business requirements dictate atomic reads and writes. ”

Some distributed systems prefer CP and in case of partitions, cluster with quorum continue to operate. other part of the cluster is either dormant/non-operational/shut down.  All the IOs are redirected to the cluster with quorum.

But there can be more than one partition in the cluster and based on the quorum policies (>50% nodes), quorum can not be established. So any IO on distributed system mail fail or it will be put into read-only mode. Availability is sacrificed in this case.

“AP – Availability/Partition Tolerance – Return the most recent version of the data you have, which could be stale. This system state will also accept writes that can be processed later when the partition is resolved. Choose Availability over Consistency when your business requirements allow for some flexibility around when the data in the system synchronizes. Availability is also a compelling option when the system needs to continue to function in spite of external errors (shopping carts, etc.)

I would highly recommend to read this IEEE article on “Consistency Tradoffs in Modern Distributed Database System Design”


PS: 1. This is my understanding. The information provided here may not be accurate or immature. Please comment if you find any misleading information.

2. All the words in the ” ” are not my wordings. They were part of the respective websites mentioned above.