code4Nix | Extract code blocks with ease

Many times, I come across scenario, wherein I need to split given file into sections and write it into different files or process them dynamically, as and when those sections are found.

I did solve this problem, although I find it to be rather inefficient, so I though of rewriting and came up with following generic function, which could be used by anyone to work on scenarios like extract function blocks from shell/perl/python scripts, or extract diff block from git diff output, etc.

Function Specification

function fn__extrt_block {

  test $# -ne 2 && exit 1
  test -f $2 && fv_iputFile=$2 || exit 2

  mkdir -p ${fv_funcWorkDir} && cd ${fv_funcWorkDir}

  grep -n "${fv_srchPtrn}" ${fv_iputFile} | cut -d':' -f1 > outfile

This grep command helps in finding out the line number to mark the beginning of block, using which following while loop will extract separate blocks.

  while true
    echo $(head -2 outfile) | read LB UB
    test ! -z ${UB} && UB=`echo ${UB} - 1 | bc`

LB and UB represents the lower and upper bound of block, i.e. beginning and ending of specific block. For instance, if the outfile has contents like this;

1:Hi Varun

15:Hi Nischal

24:Hi Team

Every iteration, reads first two lines and assigns LB to 1, UB to 15. Next line of code, decrements UB by 1, so as to mark the end of block correctly.

    echo "${LB},${UB:-\$}p" > sed_scpt
    sed -n -f sed_scpt ${fv_iputFile} > file__${LB}_${UB:-$}

Once LB and UB are set, it becomes easy to extract block from input file using sed -n, as shown above. To continue iterating, it is important to keep removing first line, after every successful iteration.

    sed '1d' outfile > outfile.n
    mv outfile.n outfile

As this is an infinite loop, it is important to break the loop, once input file is completely processed.

    test ! -s outfile && break


Let us say, you generate diff between two git commits using following command;

$> git diff versOne versTwo > ~/output__versOne_versTwo.diff

Then, execute the earlier defined function as follows;

$> fn__extrt_block '^diff' ~/output__versOne_versTwo.diff

This will generate block-specific files, with pattern file__*

GitHub Gist Code Reference

Hope this helps.

taT4Nix | Setting up Git repository in Fedora 12

I have been using Git for quite sometime now, although at work place. I could explore as much as I could, by setting up local repositories only. One day, I thought of cloning one of the repositories hosted at GitHub. However, I could not, possibly due to restrictions in place by System Administrator (I guess).

Setup GitHub Account and Installing Git

Now, to get more out of GitHub and explore further, I need a GitHub account and Linux OS. So I got started by setting up my account at GitHub and created repository, PrayogShala.

Recently, I had installed Fedora 12 (64 Bit) as Guest OS (via Virtual Box). So, I started the Virtual Machine and logged in with user account [nvarun] (created while installing the Guest OS) and executed steps as follows.

To begin contributing to repository at GitHub, one should install Git first and to do that, open up Terminal Desktop Application and refer yum steps in the reference guide.

Once dependencies are installed/updated, execute following command;

$> sudo yum install git

Once completed, find out the installed version of Git;

$> git --version
git version

Clone Repository

After installing Git, I cloned the repository earlier created on GitHub as follows;

$> git clone PrayogShala

Then, based on my experiences at work place, I added useful git aliases in $HOME/.profile and started added/updating files in the repository. The interesting thing to notice, that all the changes I make, affects repository locally. To push the changes to remote location, i.e. the repository hosted at GitHub, execute following command;

$> git push origin master

Once you do that, the Terminal prompts for username and password of GitHub account. If validated with success, the changes are pushed successfully onto GitHub.

taT4Nix | Setting up Korn Shell in Linux OS

Having spent over a year at work place, working on AIX Server, I end up doing overtime writing tools using shell scripting in Korn Shell environment. I was wondering if there is some way possible, so that I could setup similar environment at home using Linux Desktop, like Ubuntu, Fedora, etc and avoid doing overtime 🙂

AIX Server uses ksh as the default shell for every user account, whereas most of the Linux OS (I guess) have bash as the default shell. As AIX is proprietary server made by IBM, so I opted for Virtual Box instead, where I could do all my exploration.

Why Fedora 12?

Just download the ISO Image for suitable flavor of Linux OS available in the Open Source community. Thankfully, I had Fedora 12 ISO Image, which I downloaded back in 2010. To save time, I used the same to setup VM using Virtual Box. 

While installing the Fedora 12 VM (on Windows 7 Host OS), I created user account [nvarun], which by default has bash shell environment loaded whenever I open the Terminal desktop application to do command line operations.

Setting up Korn Shell

I have spent past few weeks, trying to figure out how to change default shell for user account and loading ksh environment, whenever I open Terminal desktop application and user account [nvarun] logs in by default.

  1. Open Terminal desktop application
  2. By default, the user account [nvarun] is logged in.
  3. Execute following command;
    $> grep nvarun /etc/passwd
    nvarun:x:500:500:Varun Nischal:/home/nvarun:/bin/bash
  4. As you would notice, the entry is colon-delimited and the last delimited field tells you the default shell for user account.
  5. Now to change the default shell to /bin/ksh, execute following command;
    $> sudo chsh -s /bin/ksh nvarun
    Changing shell for nvarun.
    Shell changed.
    $> exit
  6. As you exit from Terminal and re-open it again, you won’t see the change yet.

Before we proceed further, it is important to understand what AIX Server does with user accounts.

The shell uses two types of profile files when you log in to the operating system.

The shell evaluates the commands contained in the files and then runs the commands to set up your system environment.

The files have similar functions except that the /etc/profile file controls profile variables for all users on a system whereas the .profile file allows you to customize your own environment.

As we need to customize the our environment, in that case, use vi editor to edit the $HOME/.profile file and add following line;

export ENV=/etc/kshrc

The /etc/kshrc file is a shell script that customizes the Korn Shell environment and often contains list of environment variables, aliases and function definitions, etc.

Each time you start a new instance of the Korn shell, the ksh command examines the value of the ENV environment variable set in the $HOME/.profile file.

If the ENV environment variable contains the name of an existing, readable file, the ksh command runs this file as a script.

By convention, this file is named $HOME/.kshrc. You can use another name, but you must set the ENV environment variable to point to it.

Now, you may exit from Terminal and re-open it again, the changes would be reflected and this lets nvarun create aliases, functions, etc. to customize the environment further.

I am already excited to explore further, hope this helps 🙂

taT4Nix | Useful Git Aliases

Couple of months back, I mentioned that I started using Git at workplace, thanks to my colleague. So far, I have been creating local repositories at workplace and in the process, spent lot of time exploring as much as I can.

This made me use the command line extensively and I ended up writing 3-letter aliases as follows, which helped saving lot of time.

alias gbr='git branch'
alias gco='git checkout'
alias gci='git commit'
alias gcm='git checkout master'
alias glo='git log --oneline'
alias gst='git status'

This would help, only if you are using command-line interfaces, on Linux/Unix Server.

taT4Py | Extract words from Input String and Operate Functions

Working on AIX Servers with limited grep features, sometimes makes it difficult to use for particular scenarios. For instance, I want to split the lines (read from STDIN or FILE) into words, precisely. However, without grep -o option, I am clueless on how to get desired results. Since past few months, I have been investing time in learning Python and using its features to complement text processing tasks for shell scripts, I often write for automating several tasks and creating productivity tools.

Code Snippet #1

import sys, re
for line in sys.stdin.readlines():
    listofwords = [word for word in re.split('\W', line) if word]
    print listofwords

Looking at the above code snippet, four lines of code, did the trick. The features and constructs provided by Python, to implement scenarios like this, makes it look really cool. Let us understand that quickly, before we can operate functions on those words.

  1. Imports sys module for environment related capabilities and re module for regular expression capabilities.
  2. sys.stdin.readlines() reads the input from STDIN, unless EOF character has been entered.
    • After which, for loop construct iterates over the input lines, one-by-one.
  3. The special-construct present on the right-side of assignment operator, is representation of in-built function [1], filter()
    • re.split() generates [2] list of words, based on pattern given as first argument.
    • The for loop-construct iterates over the generated list and using if-construct, filtering is done.
    • This is done to make sure, there are no empty strings in the generated list using the special-construct.
    • filter(None, re.split(‘\W’, line)) can be used as an alternative, which by default takes care of empty strings, as they return false.

Code Snippet #2

What if the line contains a word, “Hi” and I want to replace all occurrences of “Hi” with “Hey”, while the list is getting generated using above approach. To make it possible, the below code snippet imports additional module string, and invokes string.replace() to do the needful.

import sys, re, string
for line in sys.stdin.readlines(): 
    listofwords = [string.replace(word, 'Hi', 'Hey') for word in re.split('\W', line) if word]
    print listofwords

This might look like not-so-useful variant, however one gets the idea to explore further and try various possibilities by themselves. Hope this helps.

In case, you feel there is any correction required, let me know and I will do the needful.


[1] [2]