Aplication Architectures using Google cloud

The below architecture diagrams helps in understanding deployment architectures for different use cases.  I want to have a quick pointer to look at these diagrams from my blog, instead of searching for them every time.

Web applications:  https://cloud.google.com/solutions/architecture/webapp

Digital asset management and sharing:  https://cloud.google.com/solutions/architecture/digitalassets

Content Management : https://cloud.google.com/solutions/architecture/contentmanagement

High Performance Computing:  https://cloud.google.com/solutions/architecture/highperformancecomputing

IOT: https://cloud.google.com/solutions/architecture/streamprocessing

Mobile Apps and Games:  https://cloud.google.com/solutions/architecture/mobileandgames

Advertisements

CAP theorem

CAP theorem states that it is impossible for a distributed system to simultaneously provide all three guarantees of

  • Consistency: All nodes see the same data at the same time.
  • Availability: a guarantee that every request receives a response about whether it was successful or failed.

Availability in CAP is defined as “every request received by a non-failing [database] node in the system must result in a [non-error] response”

  • Partition tolerance: the system continues to operate despite arbitrary message loss or failure of part of the system.

A system is partition tolerant if processing can continue in both partitions in the case of a network failure

http://robertgreiner.com/2014/08/cap-theorem-revisited/

In above article, Robert Greiner, neatly explained the CAP with diagrams.

“Given that networks aren’t completely reliable, you must tolerate partitions in a distributed system, period. Fortunately, though, you get to choose what to do when a partition does occur. According to the CAP theorem, this means we are left with two options: Consistency and Availability.”

CP – Consistency/Partition Tolerance – Wait for a response from the partitioned node which could result in a timeout error. The system can also choose to return an error, depending on the scenario you desire. Choose Consistency over Availability when your business requirements dictate atomic reads and writes. ”

Some distributed systems prefer CP and in case of partitions, cluster with quorum continue to operate. other part of the cluster is either dormant/non-operational/shut down.  All the IOs are redirected to the cluster with quorum.

But there can be more than one partition in the cluster and based on the quorum policies (>50% nodes), quorum can not be established. So any IO on distributed system mail fail or it will be put into read-only mode. Availability is sacrificed in this case.

“AP – Availability/Partition Tolerance – Return the most recent version of the data you have, which could be stale. This system state will also accept writes that can be processed later when the partition is resolved. Choose Availability over Consistency when your business requirements allow for some flexibility around when the data in the system synchronizes. Availability is also a compelling option when the system needs to continue to function in spite of external errors (shopping carts, etc.)

I would highly recommend to read this IEEE article on “Consistency Tradoffs in Modern Distributed Database System Design”

http://cs-www.cs.yale.edu/homes/dna/papers/abadi-pacelc.pdf

PS: 1. This is my understanding. The information provided here may not be accurate or immature. Please comment if you find any misleading information.

2. All the words in the ” ” are not my wordings. They were part of the respective websites mentioned above.

Python, Java and C run time.

C

  • gcc compiles your c code and generates an executable code. This code has machine instructions, means CPU can understand these instructions. Kernel has to load the executable code into memory and point CPU to starting instruction.

Java

  • java converts the java code into byte code and this byte code is executed on JVM (written in C).
  • JVM understands byte code and perform actions based on byte code instructions.

Python

  • Python compiles your code and generates a byte code.  This is not a machine code, it is python representation. This byte code is executed on Python virtual machine (PVM). PVM is the run time environment of python, it is installed as part of your python installation. This process is hidden to users who are executing .py file with python binary. Python also generates .pyc (compiled python) files  We can ship the .pyc files as your product. It might be easy to decode the python source from pyc.  python can not be used to delivery a proprietary product where they don’t want to expose alogs or logics involved.  There is a concept of frozen binaries, which packages .pyc, PVM and some support files and provide a .exe file. This is how python applications are shipped. You no need to install Python on your machine.
  • To just compile python code and create .pyc files,
python -m py_compile script.py
  • Since object code is not a machine code, Byte code instruction execution takes more time because they are not executed directly on CPU. PVM needs to understand these byte code instructions and take action.
  • There is Jpython, which converts your python program to Java byte code and run these java byte code on JVM.

Reference:

How Python Runs Programs

Choosing web framework – Django

After going through this website: https://en.wikipedia.org/wiki/Comparison_of_web_application_frameworks

I felt, man! many languages and each language has several web frameworks. This will add more confusion.  After spending 12 years with C language, I wanted to learn some high level language where applications are developed using extensively available libraries (packages/modules). I choose Python because of its wide spread usage in enterprise environments.

Python supports below frameworks. Now we have full of choices.

Project Current stable version Release date License
Bottle 0.12.8 2014-12-28[31] MIT
BlueBream 1.0 (dormant) 2011-01-18 ZPL
CherryPy 3.7.0 2015-04-24[32] BSD
CubicWeb 3.20.7 2015-04-22[33] LGPL
Django 1.8.6 2015-11-04[34] BSD
Flask 0.10.1 2013-06-14[35] BSD
Grok 2.8 (dormant) 2013-02-14[36] ZPL
Nagare 0.4.1 (dormant) 2012-01-18 BSD
Pyjs 0.8.1a (dormant) 2012-05-06 Apache
Pylons 1.0.1 (dormant) 2012-08-14 BSD
Pyramid 1.5.7 2015-04-28 BSD
TACTIC 4.3.0.v02 2015-03-31[37] EPL
Tornado 4.2 2015-05-26[38] Apache
TurboGears 2.3.5 2015-04-28[39] MIT, LGPL
web2py 2.11.2 2015-05-30[40] LGPL3
Webware 1.1.1 (dormant) 2013-01-18 Python
Zope 2 2.13.23 2015-06-29[41] ZPL

I want to choose one option out of these. After interacting with some experienced engineers in these technology and doing some study in the internet.  I came to an understanding, Flask is easy to start with if we want an application built for small scale.

“So for Small applications with simpler requirements go with FLASK. Pyramid and Django are both aimed at larger applications, but take different approaches to extensibility and flexibility. Pyramid targets flexibility and lets the developer use the right tools for their project. This means the developer can choose the database, URL structure, templating style, and more. Django aims to include all the batteries a web application will need so developers need only to open the box and start working, pulling in Django’s many modules as they go.”

“Django includes an ORM out of the box, while Pyramid and Flask leave it to the developer to choose how (or if) they want their data stored. The most popular ORM for non-Django web applications is SQLAlchemy by far, but there are plenty of other options from DynamoDB and MongoDB to simple local persistence like LevelDB or plain SQLite. Pyramid is designed to use any persistence layer, even yet-to-be-invented ones.”

“Django’s ‘batteries included’ approach makes it easy for developers who know Python already to dive in to web applications quickly without needing to make a lot of decisions about their application’s infrastructure ahead of time. Django has templating, forms, routing, authentication, basic database administration, and more built in. In contrast, Pyramid includes routing and authentication, but templating and database administration require external libraries.”

“The extra work up front to choose components for Flask and Pyramid apps yields more flexibility for developers whose use case doesn’t fit a standard ORM, or who need to interoperate with different workflows or templating systems.”

Since Django has all built in a box, as a beginner I decided to go with django. The wikipedia link compares web frameworks with following parameters:

Ajax

MVC framework

MVC push-pull

i8n & L10n

ORM

Testing framework

DB migration framework

Security framework

Template framework

Caching framework

Form Validation framwork

Will talk more about these capabilities in further blogs.

PS: Para’s with ” ” are not my wordings. I used these from https://www.airpair.com/python/posts/django-flask-pyramid

Dive into python – 1

I started reading/executing the examples given in Dive into Python 3 book by Mark Pilgrim.  After reading the first chapter I felt that I need to take some notes on the basics.  Some of the basics may be rarely remembered or talked once you make progress on coding in python. I cautiously look for those kind of points and list here.

Chapter 1

  1. Everything in Python is an object. Strings are objects. Lists are objects. Functions are objects. Classes are objects. Class instances are objects. Even modules are objects.

    My first thought was, What?. The example given in the book was:

    All functions have a built-in attribute __doc__, which returns the docstring defined in the function’s source code. The sys module is an object which has (among other things) an attribute called path.

    You may have heard the term “first-class object” in other programming contexts. In Python, functions are first-class objects. You can pass a function as an argument to another function. Modules are first-class objects. You can pass an entire module as an argument to a function. Classes are first-class objects, and individual instances of a class are also first-class objects.


  2. Every function deserves a decent docstring.
  3.  All names in Python are case-sensitive: variable names, function names, class names, module names, exception names. If you can get it, set it, call it, construct it, import it, or raise it, it’s case-sensitive.
  4. The below statement is like ternary operator in C (?:)
    multiple = 1024 if a_kilobyte_is_1024_bytes else 1000
  5. In Python, variables are never explicitly typed. Python figures out what type a variable is and keeps track of it internally.
  6. Python allows function arguments to have default values; if the function is called without the argument, the argument gets its default value. This means the argument is optional;
  7. You can also pass values into a function by name. These are received as key-word arguments. If you pass just a value they are recieved as positional arguments. Positional arguments should be passed in the order. We should pass them first and then the named arguments. Reading the argument list from left to right, once you have a single named argument, the rest of the arguments must also be named.
  8.  i = 10, In python, ‘i’ is a name with a binding to the object created by interger 10. In C langauge,  ‘i’ is pointing to a memeory location where the integer value 10 is stored. C developers has to keep this point in mind.

Some famous security vulnurabilities

1. RSA signature forgery by unprivileged users.

RSA signatures are used to authenticate the source of a message. To prevent RSA signatures from being forged, messages are padded with data to ensure message hashes are adequately sized. One such padding scheme is specified in the Public-Key Cryptography Standard #1 (PKCS-1), which is defined in RFC 3447. Many RSA implementations may fail to properly verify signatures. Specifically, the verifier may incorrectly parse PKCS-1 padded signatures, ignoring data at the end of a signature. If this data is ignored and a RSA key with a public exponent of three is used, it may be possible to forge the signing key’s signature.

Note that any application that uses RSA signatures may be affected by this vulnerability. This includes, but is not limited to, SSH, SSL, PGP, and X.509 applications.

http://en.securitylab.ru/notification/276087.php

http://www.mail-archive.com/cryptography@metzdowd.com/msg06732.html

http://www.mail-archive.com/cryptography@metzdowd.com/msg06537.htm

http://en.securitylab.ru/notification/276087.php

2. Telnet login vulnerability gives root access to unauthorized users.

/usr/src/cmd/cmd-inet/usr.sbin/in.telnetd.c
3198
3199     } else /* default, no auth. info available, login does it all */ {
3200                   (void) execl(LOGIN_PROGRAM, “login”,
3201                                   “-p”, “-h”, host, “-d”, slavename,
3202                                   getenv(“USER”), 0);
3203     }

/usr/src/cmd/login/login.c
1397                            break;
1398
1399                  case ‘f’:
1400                            /*
1401                             * Must be root to bypass authentication
1402                             * otherwise we exit() as punishment for trying.
1403                             */
1404 if (getuid() != 0 || geteuid() != 0) {
1405           audit_error = ADT_FAIL_VALUE_AUTH_BYPASS;
1406
1407           login_exit(1);     /* sigh */
1408           /*NOTREACHED*/
1409 }
1410 /* save fflag user name for future use */
SCPYL(user_name, optarg);
1411
1412 fflag = B_TRUE;

So if we supply a USER environment variable of “-f<username>” we can get in without a
password.

3. DNS server cache poisoning by unprivileged users

http://unixwiz.net/techtips/iguide-kaminsky-dns-vuln.html

http://www.infobyte.com.ar/demo/evilgrade.htm

4.Cross‐site scripting attacks, which allow attackers to execute scripts within the context of a user’s browser.

https://www.governmentsecurity.org/SecurityHackingNews/Sun_Alert_247046_Cross_Site_Scripting_XSS_Vulnerability_in_Sun_Management_Center_SunMC_Performance_Reporting_Module

5.Injection flaws, such as SQL injection attacks in which SQL commands are sent as
part of input data.

6. Poorly managed authentication in distributed applications that allow, for example, a
victim’s username and passwords to be stolen

7. Insecure communications, in which private and confidential information is sent in
unencrypted or easily decrypted form

Secure programming Tips

Some of the programming  mistakes lead to serious security issues (vulnerabilities). This programming errors very important to deal with because these costs companies a lot in terms of money, customer confidence on product. Some of these may cause

-> threaten the confidentiality of private information

-> threaten the integrity of data and operations

->threaten to disrupt availability of critical systems

-> Escalate the privileges of unauthorized users

1. Check for integer overflow.

While calculating Fibanoci seris or factorial, the next value may exeed the MAX interger value, make sure to check for the overflow condition.

2. Create temporary file with mkstemp(3C) mkdtemp(3C)

Most common progrmming mistake is creating temporary file with a well known names or predictable names like tmp.$$ or tmp.pid. If a malicious user predicts this temp file pattern, we will create them in advance and try to manage them with his own data or cause application crash.

3. Buffer overflow issues with string operations.

people say use strn series functions for string operations.The strncpy() functions are preferable to strcpy() because they accept boundaries for buffers that can be checked against. However, they are still vulnerable to certain attacks if used improperly:

a. passing of NULL for src or dest causes exception
b. ‘count’ size parameter is often incorrectly passed in
c. not guaranteed to have null terminated string upon exit

Make sure the buffer and bounds are the proper size to hold the source string plus a NULL character.

4.Improper Input Validation

5. Web application security : For more information about application security, especially Web
applications, see the Open Web Application Security Project (OWASP) at http://www.owasp.org.

http://nob.cs.ucdavis.edu/~bishop/secprog/index.html

http://sunsite.uakom.sk/sunworldonline/swol-08-1998/swol-08-security.html