Make file example

The make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them.

Most of the engineers who started working on projects in the middle or supporting the product which are in maintenance phase has less visibility to Make files. For them, make files are of least concern. I want to present Make file basics with simple examples here. I was in this situation in my previous job.  I am into software development for almost 5 years,   would like to explain the basics of makefile I learnt while creating new projects.

In any Make file, we see mainly these sections: Variable, targets.

Variable syntax:

NAME = value

Target syntax:

target … : prerequisitesrecipe
        …
        …

A target is usually the name of a file that is generated by a program.

A prerequisite is a file that is used as input to create the target.

A recipe is an action that make carries out. A recipe may have more than one command, either on the same line or each on its own line.

By default, Makefile targets are “file targets” – they are used to build files from other files. Make assumes its target is a file, and this makes writing Makefiles relatively easy.

However, sometimes you want your Makefile to run commands that do not represent physical files in the file system. Good examples for this are the common targets “clean” and “all”.These special targets are called phony and you can explicitly tell Make they’re not associated with files. In terms of Make, a phony target is simply a target that is always out-of-date, so whenever you ask make clean, it will run, independent from the state of the file system

.PHONY: clean all product-daemon subproduct-daemon iotest framework metastore

The below is an example of a makefile of a top directory in a product:

TARGET=production

include makefiles/$(TARGET).make
export CC
export CC_PERMISIVE
export CXX
export CXX_PERMISIVE
export OPTFLAGS
export CXXFLAGS
export LDFLAGS

PROTOC = protoc
GRPC_CPP_PLUGIN = grpc_cpp_plugin
GRPC_PYTHON_PLUGIN = grpc_python_plugin
GRPC_CPP_PLUGIN_PATH ?= `which $(GRPC_CPP_PLUGIN)`
GRPC_PYTHON_PLUGIN_PATH ?= `which $(GRPC_PYTHON_PLUGIN)`

PROTOS_PATH = ./protos
PROTOS_OUTPUT_CPP = ./protos/src
PROTOS_OUTPUT_PYTHON = ./build/python

PROTO_FILES = product_daemon_rpc.proto \
 subproduct_daemon_rpc.proto \
 metadb_kv.proto

PROTO_GEN_HEADERS = $(patsubst %,$(PROTOS_OUTPUT_CPP)/%, $(PROTO_FILES:.proto=.pb.h) $(PROTO_FILES:.proto=.grpc.pb.h))
PROTO_GEN_SOURCES = $(patsubst %,$(PROTOS_OUTPUT_CPP)/%, $(PROTO_FILES:.proto=.pb.cc) $(PROTO_FILES:.proto=.grpc.pb.cc))

all: common/common.a framework/framework.a meta-store/metastore.a product-daemon subproduct-daemon iotest

common/common.a:
 cd common && $(MAKE)

framework/framework.a:
 cd framework && $(MAKE)

meta-store/metastore.a: $(PROTO_GEN_HEADERS) $(PROTO_GEN_SOURCES)
 cd meta-store && $(MAKE) STATS=1

clean:
 cd common && $(MAKE) clean
 cd framework && $(MAKE) clean
 cd meta-store && $(MAKE) clean
 cd subproduct-daemon && $(MAKE) clean
 cd product-daemon && $(MAKE) clean
 cd utils/iotest && $(MAKE) clean
 rm -f $(PROTOS_OUTPUT_CPP)/*.cc
 rm -f $(PROTOS_OUTPUT_CPP)/*.h
 rm -f $(PROTOS_OUTPUT_CPP)/*.d
 rm -f $(PROTOS_OUTPUT_PYTHON)/*.py

$(PROTOS_OUTPUT_CPP)/%.pb.h $(PROTOS_OUTPUT_CPP)/%.pb.cc: $(PROTOS_PATH)/%.proto
 $(PROTOC) -I $(PROTOS_PATH) --cpp_out=$(PROTOS_OUTPUT_CPP) $<

$(PROTOS_OUTPUT_CPP)/%.grpc.pb.h $(PROTOS_OUTPUT_CPP)/%.grpc.pb.cc: $(PROTOS_PATH)/%.proto
 $(PROTOC) -I $(PROTOS_PATH) --grpc_out=$(PROTOS_OUTPUT_CPP) --plugin=protoc-gen-grpc=$(GRPC_CPP_PLUGIN_PATH) $<

product-daemon: $(PROTO_GEN_HEADERS) $(PROTO_GEN_SOURCES) common/common.a framework/framework.a meta-store/metastore.a
 cd product-daemon && $(MAKE)

subproduct-daemon: $(PROTO_GEN_HEADERS) $(PROTO_GEN_SOURCES) common/common.a framework/framework.a
 cd subproduct-daemon && $(MAKE)

iotest:
 cd utils/iotest && $(MAKE)

framework: framework/framework.a

metastore: metastore/metastore.a

.PHONY: clean all product-daemon subproduct-daemon iotest framework metastore
$ cat makefiles/production.make
CC=gcc
CC_PERMISIVE=gcc
CXX=g++
CXX_PERMISIVE=g++

OPTFLAGS=-g3 -ggdb -O3 -DNDEBUG
  • include makefiles/$(TARGET).make

    The include directive tells make to suspend reading the current makefile and read one or more other makefiles before continuing.

  • export CC 
    export CC_PERMISIVE 
    export CXX
    export CXX_PERMISIVE
    export OPTFLAGS
    export CXXFLAGS
    export LDFLAGS

    When make runs a recipe, variables defined in the makefile are placed into the environment of each shell. This allows you to pass values to sub-make invocations . By default, only variables that came from the environment or the command line are passed to recursive invocations. You can use the export directive to pass other variables. To pass down, or export, a variable, make adds the variable and its value to the environment for running each line of the recipe. The sub-make, in turn, uses the environment to initialize its table of variable values.

  • GRPC_CPP_PLUGIN_PATH ?= which $(GRPC_CPP_PLUGIN)

This is called a conditional variable assignment operator, because it only has an effect if the variable is not yet defined. This is equivalent to

ifeq ($(origin GRPC_CPP_PLUGIN_PATH), undefined)
  GRPC_CPP_PLUGIN_PATH = which $(GRPC_CPP_PLUGIN)
endif
  • PROTO_GEN_HEADERS = $(patsubst %,$(PROTOS_OUTPUT_CPP)/%,
     $(PROTO_FILES:.proto=.pb.h) $(PROTO_FILES:.proto=.grpc.pb.h))

A substitution reference substitutes the value of a variable with alterations that you specify. It has the form ‘$(var:a=b)’ (or ‘${var:a=b}’) and its meaning is to take the value of the variable var, replace every a at the end of a word with b in that value, and substitute the resulting string.

For example:

foo := a.o b.o c.o
bar := $(foo:.o=.c)

sets ‘bar’ to ‘a.c b.c c.c’. This case is equivalent to ‘$(patsubst %.o,%.c,$(foo))’

  • meta-store/metastore.a: $(PROTO_GEN_HEADERS) $(PROTO_GEN_SOURCES)
     cd meta-store && $(MAKE) STATS=1

    For generating target meta-store/metastore.a , make has to make sure the dependencies $(PROTO_GEN_HEADERS) and $(PROTO_GEN_SOURCES) are available. The recipe is “cd meta-store && make STATS=1”, This sub-make gets all the exported variables in its environment.

  • $(PROTOS_OUTPUT_CPP)/%.pb.h $(PROTOS_OUTPUT_CPP)/%.pb.cc: $(PROTOS_PATH)/%.proto
     $(PROTOC) -I $(PROTOS_PATH) --cpp_out=$(PROTOS_OUTPUT_CPP) $<

    Target is: A rule with multiple targets is equivalent to writing many rules, each with one target, and all identical aside from that. The same recipe applies to all the targets. In above case for each maching target, it makes sures that the corresponding prerequisite exits and corresponding rule is executed.

    $(PROTOS_OUTPUT_CPP)/%.pb.h $(PROTOS_OUTPUT_CPP)/%.pb.cc
  • Dependency:
    $(PROTOS_PATH)/%.proto

    Recipe:

    $(PROTOC) -I $(PROTOS_PATH) --cpp_out=$(PROTOS_OUTPUT_CPP) $<

    Make has some special variables called, automatic variables. These variables have values computed afresh for each rule that is executed, based on the target and prerequisites of the rule. The widely used ones are:

    • $@ The file name of the target of the rule.
    • $%The target member name.

Example1:

foolib(hack.o) : hack.o
        ar cr foolib hack.o

foo.a(bar.o) then ‘$%’ is bar.o and ‘$@’ is foo.a. ‘$%’ is empty when the target is not an archive member.

  • $< The name of the first prerequisite.
  • $^The names of all the prerequisites

Example2:

all: library.cpp main.cpp

In this case:

  • $@ evaluates to all
  • $< evaluates to library.cpp
  • $^ evaluates to library.cpp main.cpp

 

common directory  makefile:

CXX?=g++
CC?=gcc
ASM?=g++
OPTFLAGS?=-g3 -ggdb -O0

PACKAGES=libconfig++ libglog
LIBS=-pthread -lgtest

CXXFLAGS+=-std=c++14 -MMD -Wall -I. -I../init -I/usr/local/include `pkg-config --cflags ${PACKAGES}` $(OPTFLAGS)
CFLAGS+=-MMD -Wall $(OPTFLAGS)
LDFLAGS+=-L/usr/local/lib `pkg-config --libs ${PACKAGES}` ${LIBS}

ASM_SRC=codeword_utils.o
CRC_SRC=crc.o

OBJECTDIR=build

ASM_OBJS=$(patsubst %,$(OBJECTDIR)/%,$(ASM_SRC))
CRC_OBJS=$(patsubst %,$(OBJECTDIR)/%,$(CRC_SRC))
VPATH+=.

SRC=$(CRC_OBJS) $(ASM_OBJS)

all: codeword_utils mpmcq

codeword_utils: $(CRC_OBJS) $(ASM_OBJS)
 ar rcs common.a $(SRC)

mpmcq:
 cd mpmcq && $(MAKE)

$(OBJECTDIR)/%.o: %.cpp
 $(CXX) $(CXXFLAGS) -c -o $@ $<

$(OBJECTDIR)/%.o: %.s
 $(ASM) $(CXXFLAGS) -c -o $@ $<

-include $(OBJECTDIR)/*.d

clean :
 rm -f $(OBJECTDIR)/*.o $(OBJECTDIR)/*.d common.a codeword_utils
 cd mpmcq && $(MAKE) clean

.PHONY: clean all codeword_utils mpmcq

most of the above sections are already discussed. The new ones are:

  • VPATH+=.

    The value of the make variable VPATH specifies a list of directories that make should search. Most often, the directories are expected to contain prerequisite files that are not in the current directory; however, make uses VPATH as a search list for both prerequisites and targets of rules.

  • -include $(OBJECTDIR)/*.d

    This acts like include in every way except that there is no error (not even a warning) if any of the filenames (or any prerequisites of any of the filenames) do not exist or cannot be remade.

meta-store makefile

CXX?=g++
CC?=gcc
CXX_PERMISIVE?=g++
OPTFLAGS?=-g3 -ggdb -O0

PACKAGES=libconfig++

CXXFLAGS+=-std=c++14 -MMD -Wall -I. -I/usr/local/include -Iapi -Istore -Icluster -Itransactions -Igen-cpp2 -I../protos/src `pkg-config --cflags ${PACKAGES}` $(OPTFLAGS)
CXXFLAGS+= -I transactions/meta_keys -Itransactions/sql -Itransactions/key-value

ifeq ($(STATS), 1)
CXXFLAGS+= -D_ENABLE_STATS_
endif

CFLAGS+=-MMD -Wall $(OPTFLAGS)

SRCPROTOS = metadb_kv.pb.o

VPATH+=api::store::cluster::transactions::tests::gen-cpp2
VPATH+=../protos/src
VPATH+=transactions/sql \
 transactions/key-value

SRC = meta1.o \
 meta2.o \
 

THRIFT_CPP_FILES = $(wildcard gen-cpp2/*.cpp)
THRIFT_CPP_FILES := $(filter-out gen-cpp2/non-shared_constants.cpp, $(THRIFT_CPP_FILES))
SRC += $(patsubst gen-cpp2/%.cpp,%.o,$(THRIFT_CPP_FILES))

SRC += $(SRCPROTOS)

OBJECTDIR=build

OBJS=$(patsubst %,$(OBJECTDIR)/%,$(SRC))

all: db test

db: $(OBJS)
 ar rcs metastore.a $(OBJS)

test: db
 cd tests && $(MAKE)

$(OBJECTDIR)/%.o: %.cpp
 $(CXX) $(CXXFLAGS) -c -o $@ $<

$(OBJECTDIR)/%.o: %.cc
 $(CXX) $(CXXFLAGS) -c -o $@ $<

-include $(OBJECTDIR)/*.d

clean :
 rm -f $(OBJECTDIR)/*.o $(OBJECTDIR)/*.d
 rm -f metastore.a
 cd tests && $(MAKE) clean

.PHONY: clean all db test
  • THRIFT_CPP_FILES = $(wildcard gen-cpp2/*.cpp)

    Wildcard expansion does not happen when you define a variable.However, if you use the value of objects in a target or prerequisite, wildcard expansion will take place there. To do the wildcard expansion we use the keyword wildcard.

Race condition Scenario 1

Recently we hit an issue where the client program has stuck. The client has been written in such way it sends requests to server in synchronous manner (sends the next request only after receiving the acknowledgement for current request).  This issue happens intermittently.  This symptom tells that there is some race condition.

On the server side, there are two threads

thread 1

  1. submits a request to one of its internal submission queue.
  2. increments the io_submitted value.

thread 2

  1. picks up the item from submission queue and does an asynchronous IO using libaio.
    1. libaio thread calls the call-back function passed to it once the IO is done.
    2. As part of call-back function aio thread enqueues the request into completion queue
  2. picks up the item from completion queue and increments io_completed value.
  3. the does a check io_submitted == io_completed to do next set of task.
  4.  after completing the next set of tasks, sends a response to the client.

The problem is that the client is not receiving the acknowledgment.  Why?

There is a race:   Before thread 1 increments the io_submitted value,  thread2 increments io_completed and does a comparison check. This can be possible if thread1 is scheduled out before we increment io_submitted value.

Couple of solutions:

  1. Move increment before submitting request o internal submission queue.
  2. Use spin lock to protect the io_submitted

 

 

Working with Docker

Docker is a orchestration layer on top of Linux containers. It creates a lightweight work environment like BSD jails, Solaris zones. You need to install set of packages in your ubuntu OS:

https://docs.docker.com/engine/installation/linux/ubuntulinux/

1. Create docker images from docker file
docker build -t image1 . ( Docker file is located at ./)

2. List out docker images

docker images

3. changes to images and committing changes

docker run -i -t –name guest image1:latest /bin/bash
apt-get install vim
docker stop guest
docker commit -m “message” -a “author” <container-id of guest> image1:latest

docker rm guest

docker run -i -t –name guest image1:latest /bin/bash

check if installed vim is installed.

dpkg –get-selections |grep vim
3. Update existing image
docker pull mysql
docker stop my-mysql-container
docker rm my-mysql-container
docker run –name=my-mysql-container –restart=always \
-e MYSQL_ROOT_PASSWORD=mypwd -v /my/data/dir:/var/lib/mysql -d mysql

your data should have been stored on -v volumes.

4. List containers

docker ps -a

5. Stop container

docker stop guest

6. start container

docker start guest

7. Aattach a running container, you exited from the session, but container still running

docker attach guest

8. Remove a container

docker rm guest

9. Reomve docker images

docker rmi $(docker images -f “dangling=true” -q)

 

In general,  egnieers use a script to start a container and share the directories from the host machine. Ex:-

rundocker.sh

#!/bin/sh

mkdir -p $HOME/$1-shared

docker run –rm -t -i –name $1 -v $HOME/.gitconfig:/root/.gitconfig -e SSH_AUTH_SOCK=$SSH_AUTH_SOCK -e DISPLAY=$DISPLAY -v /tmp/.X11-unix:/tmp/.X11-unix:rw –privileged –net=host -v $HOME/.Xauthority:/root/.Xauthority -v $HOME/$1-shared:/root/shared  image1:latest /bin/bash

option description:

–rm  : remove container when it exit.

–name : name of the container

-i : keep STDIN even container is not attached.

-t : allocate a psudo TTY.

-v :  directory mapping from host to container (sharing of host directories)

Git Commands (Every day use)

Here are the list of git commands that I use every day:

  • Clone remote repository
  • list the branches in test-repo repository.
    • git branch -vv
  • Switch between branches
    • git checkout milestone1
    • git checkout milestone2
  • Create a new branch work1 from milestone1
    • git checkout milestone1
    • git branch work1
    • git checkout work1
    • You can combine above two commands with “git checkout -b work1”
  • Push new branch work1 to repository test-repo
    • git push origin
  • Merging branches (merge milestone2 code to work1)
    • git checkout milestone2
    • git pull (git fetch & git merge origin/milestone2)
    • git checkout work1
    • git pull
    • git merge –no-ff milestone2
    • git push origin work1
  • push a modification to file1 to work1 branch
    • git chekcout work1
    • git pull
    • git add file1
    • git commit -m “message” (or git commit -a -m “message” to skip staging area)
    • git push origin work1
  • Delete local branch
    • git branch -d work1 (to delete tracking branch)
    • git push origin  :work1 (to delete branch on remote repository)
  • You have done some changes in your local tracking branch, but you want to work on different branch and come back. If you have new file, then add it to the index.
    • git add newfile
    • Do changes in current indexed file file1.
    • git stash
    • git checkout master
    • do some work on master branch
    • git checkout work1
    • git stash pop (will pop both file1 and newfiles changes on top of work1 branch)
  • See the history of commits
    • git log
    • git log -p -1
    • git log –stat
  • Code differences
    • git diff origin/master (local uncommitted changes with remote tracking branch)
    • git diff master (local uncommitted changes with tracking branch, where you commits your changes)
    • git diff COMMIT^ COMMIT (compare commit with ancestor commit of tracking branch)
    • git diff –staged (staged changes with tracking branch recent commit)
    • git diff (not committed)
    • git diff HEAD (upstaged changes with recent commit)
    • git show (to see the commited changes, difference with parent commit)
    • git diff milestone1..milestone2 (difference between two branches)
  • Status of your changes
    • git status
  • Resetting local committed changes
    • git reset <commit-id> (HEAD is moved to the specified commit-id)
  • Resetting local staged changes
    • git rm –cached
  • Discard changes in current working directory. (not staged)
    • git checkout — file1
  • Bringing changes to exiting commit
    • git add file2
    • git commit –amend (will add file2 change into same commit, not pushed)
  • Generate patch from commit 
    • git format-patch -n1
    • git am <p (apply a specific patch)

We need to understand some theory behind how git works. Some common terms we come across while working with git:

Repository, Remote tracking branch,  Tracking branch, Local branch

Repository – A set of branches related to project. (git clone <repository link>)

Remote tracking branch –

Remote-tracking branches are references to the state of remote branches. They’re local references that you can’t move; they’re moved automatically for you whenever you do any network communication. Remote-tracking branches act as bookmarks to remind you where the branches in your remote repositories were the last time you connected to them.

On my computer origin/master is the remote tracking branch which refers to the branch on hosting computer. When I do git pull, git push,  the git command uses origin/master info to talk to hosting computer(github.com).

Tracking branch /Local branch – This is the branch on my computer, which will have all the commits I have done but not yet pushed to the remote branch.

git branch -vv
  ganga     8e22413 [origin/ganga: ahead 2] integration of allocation logic
  master    4ae3a25 [origin/master] db change
* bug1 c8676f9 [dev/bugs: ahead 4, behind 2] fix for illegal option
  fet1   62a363b container changes

In above example,  Local branch or tracking branches are : ganga, master, bug1 and testing

Remote tracking branches : origin/ganga, origin/master, dev/bugs,

Tracking branches are local branches that have a direct relationship to a remote branch. If you’re on a tracking branch and type git pull, Git automatically knows which server to fetch from and branch to merge into

Git index (aka staging area/cache/directory cache/staged files) – Changes that are added for commit.

 

Good string programming puzzle

Problem Definition:

You are challenged to write an algorithm to check if a given string, s, can be formed from two other strings, part1 and part2.

The restriction is that the characters in part1 and part2 are in the same order as in s.

Test Cases:

  1. s = “Bananas from Bahamas” ;  part1 =”Bahas” ; part2 =” Bananas from am”
  2. s =”Can we merge it? Yes, we can!”; part1=” an mrgei? e,we”; part2=”C wee tYs can!”
  3. s=”Making progress; part1=”Mak pross”; part2=”inggre”
  4. s=’codewars’; part1=’code’; part2=’wars’
  5. s=’codewars’; part1=’cdw’; part2=’oears’

 

Logic:

  1. Start matching character by character, there are two possibilities to start
    1. ‘s’ with ‘part1’
    2. ‘s’ with ‘part2’
  2. ‘s’ with part1:  Start matching string ‘s’ in part1 as long it matches and then move to part2 as long as it matches there. come back to part1 and try to match as much it matches then move to part2 … till the end of string ‘s’. If we can’t make progress then return False.
  3. ‘s’ with part2 :  If step2 fails, then start matching ‘s’ with part2…
  4. It is not just two possibilities, for every character match there are two possibilities, so we need to handle all those cases.
  5. For example:
    1. s = “Bananas from Bahamas” ;  part1 =”Bahas” ; part2 =” Bananas from am”
  6. For first character ‘B’ from s,  two possibilities s[0] == p1[0], s[0] == p2[0].  ‘s’ matches with ‘p1’  till “Ba”. The  rest of ‘s’ (“nanas from Bahamas”) does not match with p2 (“Bananas from am”) . so we have try with second possibility, s[0] == p2[0], this matches till “Bananas from ” in p2.  The rest of ‘s’ (“Bahamas”)  does not match with p2 (“am”). Then try to match “Bahamas” with p1 (“Bahas”), p2(“am”).
  7.    s=”Bahamas”, p1=”Bahas”, p2=”am”.  comparisons should go like this:
          'B' in s and 'B' in p1 (1)
               'a' in s and 'a' in p1   (2)
                  'h' in s and 'h' in p1  (3)
                      'a' in s and 'a' in p1 (False) 
                         'm' in s and 's' in p1 (False) 
                         'm' in s and 'a' in p2 (False) 
                      'a' in s and 'a' in p2 (True)  (4)
                         'm' in s and 'a' in p1 (False)
                         'm' in s and 'm' in p2  
                            'a' in s and 'a' in p1
                               's' in s and 's' in p1
                                  '' in s and '' in p1 (True) (5)
                                  

True value is bubbles up to (1) . In the order of 5,4,3,2,1

Python code:

Recursive solution:

def is_merge(s, part1, part2):
  if not part1:
     return s == part2
  if not part2:
     return s == part1
  if not s:
    return part1 + part2 == ''
  if s[0] == part1[0] and is_merge(s[1:], part1[1:], part2):
    return True
  if s[0] == part2[0] and is_merge(s[1:], part1, part2[1:]):
    return True
  return False

Iterative Solution:

def is_merge(s, part1, part2):
    queue = [(s,part1,part2)]
    while queue:
        str, p1, p2 = queue.pop() 
        if str:
           if p1 and str[0] == p1[0]:
              queue.append((str[1:], p1[1:], p2))
           if p2 and str[0] == p2[0]:
              queue.append((str[1:], p1, p2[1:]))
        else:
           if not p1 and not p2:
              return True
    return False

Aplication Architectures using Google cloud

The below architecture diagrams helps in understanding deployment architectures for different use cases.  I want to have a quick pointer to look at these diagrams from my blog, instead of searching for them every time.

Web applications:  https://cloud.google.com/solutions/architecture/webapp

Digital asset management and sharing:  https://cloud.google.com/solutions/architecture/digitalassets

Content Management : https://cloud.google.com/solutions/architecture/contentmanagement

High Performance Computing:  https://cloud.google.com/solutions/architecture/highperformancecomputing

IOT: https://cloud.google.com/solutions/architecture/streamprocessing

Mobile Apps and Games:  https://cloud.google.com/solutions/architecture/mobileandgames