taT4Py | Recursively Search Regex Patterns

[UPDATE: 09/28/2014]

I have mainly used python for text parsing, validation and transforming as needed. If it was done using shell script, I would end up writing variety of regular expression to play around.

Getting Started

Well, python is no different and in order to cook up regular expressions, one must import re (module) and get started.

import re

So far, I have been able to use the patterns exactly the same way as I would with grep or sed. Usually, I end up writing multiple search patterns, as the script evolves. While using python, I find it intuitive to create dictionary of compiled search patterns (RegexObject), I wrote to style unified differences, as follows.

regexDict = {
    'HEADER': re.compile("^@@ ([+-][0-9]+(,[0-9]+)? ?){1,2} @@$"),
    'ADD':    re.compile("^\+"),
    'DEL':    re.compile("^\-")

Search Recursively

Looking at the above dictionary, there are only 3 key-value pairs, so writing if-else construct would be easy. Let say, such a dictionary is dynamically created and can have any number of key-value pairs.

All you need to figure out, whether data matches particular search pattern or not. If yes, print the data, or transform the data, etc. In this post, we will go one step further and redesign the if-else construct used to style unified differences, as follows.

with open(inputFile, 'r') as fileObj:

    ### Using Slice To Ignore First 2 Lines
    for line in fileObj.read().splitlines()[2:]:

        fn__recurSearch(regexDict.keys(), line)

The for-statement invokes function with two arguments, first one of the type – iterator and the other one – string. Let us look at the function definition below;

def fn__recurSearch(iterator, data):

    if not iterator:
        print data

    key = iterator.pop(0)

    matchObj = regexDict[key].search(data)

    if not matchObj:
        fn__recurSearch(iterator, data)

    print data

The function looks straight forward, however this would simply dump the file as is on the STDOUT. If you have paid attention, you will notice, we have not added logic to style unified differences !! Well, that’s an exercise left it for you, otherwise I will try to cover next time.

[UPDATE: 09/28/2014]

Let us add the logic to style unified differences, as follows;

def fn__recurSearch(iterator, data):

    global codeChunkList

    if not iterator:
        ##print data

    key = iterator.pop(0)

    matchObj = regexDict[key].search(data)

    if not matchObj:
        fn__recurSearch(iterator, data)

    if key == 'HEADER':
        if codeChunkList:
            print trHTMLCode % '<br />\n'.join(codeChunkList)
            codeChunkList = []

    ##print data

Indeed, the recursive function does make it look so easy, yet simple. The output produced by new design, yields the same results as did the old one.

WAIT, there’s more..

How about styling context differences ?? This approach can have multiple applications, depending on your problem scenario. Give it a try, feel free to share your thoughts..

taT4JS | Handle Select Menu Inter-Dependencies

Recently, while prototyping intranet website I came across scenario, wherein the top section of the web page is required to have two drop-down lists, i.e. select tags, which would have inter-dependency.

Let say, the web page is divided into 3 sections;

  • #top, having drop-down lists,
  • #middle, which will reload asynchronously based on the context set by drop-down lists,
  • #bottom, like page footer.

The first drop-down list is populated with names of programming languages and second drop-down list is populated with list of categories like Tutorials, Tips and Tricks, FAQs, etc.

Playing Tricks

$('#selMenu_typeOne').change(function() {
$('#selMenu_typeTwo').change(function() { 
        //execute some function, or 
        //do some global variable assignment operations

As per the above snippet, as soon as user selects an item from #selMenu_typeOne drop-down list, it would automatically trigger the similar event for the #selMenu_typeTwo drop-down list.

To implement the async functionality, I used $.getJSON() functions to read necessary information from JSON file, whose naming is dependent on the items selected from drop-down lists and format it into tables, lists, etc.

Tips to Remember

Let say, user selected Java from first list, which resets the second list to select Tutorials by default. This would, in turn cause #middle section to reload async accordingly. This enables user to change the selection in any of the drop-down lists to view data asynchronously with ease.

taT4Py | Convert AutoSys Job Attributes into Python Dictionary

[UPDATE: 09/28/2014]

If you ever look at the definition of specific AutoSys Job, you would find that it contains attribute-value pairs (line-by-line), delimited by colon ‘:’ I thought it would be cool to parse the job definition, by creating python dictionary using the attribute-value pairs.

Let us take a look at sample job definition;

$> cat sample_jil
insert_job: A0001
command: echo "Hi"
condition: s(B0001, 03\:00) & v(SRVR) = "UP"
std_out_file: >/home/nvarun/outfile
std_err_file: >/home/nvarun/errfile
group: NV

Getting Started

To convert this into Python Dictionary, execute ignore the following command;

$> sed "s/^\([^:]*\):\(.*\)$/'\1':'\2'/" sample_jil > sample_pydict
$> cat sample_pydict
'insert_job':' A0001
'command':' echo "Hi"'
'condition':' s(B0001, 03\:00) & v(SRVR) = "UP"'
'std_out_file':' >/home/nvarun/outfile'
'std_err_file':' >/home/nvarun/errfile'
'group':' NV'

We are half-way through, to complete the conversion, write following steps in python script and populate dictionary as follows;

import string
jobDefn = {}
with open('sample_pydict', 'r') as f:
    for line in f.read().splitlines():
        colon = string.find(line, ':')
        key = line[:colon]     ##string.replace(line[:colon], "'", "")
        val = line[colon+1:].strip()
        jobDefn[key] = val        
print jobDefn

[UPDATE: 09/28/2014]

As per Antonio’s comments, one can optimize the code as follows, by ignoring sed as well;

import string
jobDefn = {}
with open('sample_pydict', 'r') as f:
    for line in f:
        key, val = line.split(':')
        jobDefn[key] = val.strip()
print jobDefn

However, there are chances when the values might contain ‘:’ as well, you could switch back to the earlier solution. Otherwise, invoking split() as above, throws ValueError: too many values to unpack.


  1. Using f.read() reads the input file at one go and invoking splitlines() splits the input into list of several lines, resulting in creating an iterator.
  2. The for-statement iterates over each line from file object wherein position of first occurrence of colon is found and used for extracting key, value based on slicing and invokes split() to determine key, value.
  3. At the end of the loop, the dictionary object jobDefn is printed.

Hope this helps.

taT4Nix | Setting up Git repository in Fedora 12

I have been using Git for quite sometime now, although at work place. I could explore as much as I could, by setting up local repositories only. One day, I thought of cloning one of the repositories hosted at GitHub. However, I could not, possibly due to restrictions in place by System Administrator (I guess).

Setup GitHub Account and Installing Git

Now, to get more out of GitHub and explore further, I need a GitHub account and Linux OS. So I got started by setting up my account at GitHub and created repository, PrayogShala.

Recently, I had installed Fedora 12 (64 Bit) as Guest OS (via Virtual Box). So, I started the Virtual Machine and logged in with user account [nvarun] (created while installing the Guest OS) and executed steps as follows.

To begin contributing to repository at GitHub, one should install Git first and to do that, open up Terminal Desktop Application and refer yum steps in the reference guide.

Once dependencies are installed/updated, execute following command;

$> sudo yum install git

Once completed, find out the installed version of Git;

$> git --version
git version

Clone Repository

After installing Git, I cloned the repository earlier created on GitHub as follows;

$> git clone https://github.com/nvarun/PrayogShala.git PrayogShala

Then, based on my experiences at work place, I added useful git aliases in $HOME/.profile and started added/updating files in the repository. The interesting thing to notice, that all the changes I make, affects repository locally. To push the changes to remote location, i.e. the repository hosted at GitHub, execute following command;

$> git push origin master

Once you do that, the Terminal prompts for username and password of GitHub account. If validated with success, the changes are pushed successfully onto GitHub.

taT4Nix | Setting up Korn Shell in Linux OS

Having spent over a year at work place, working on AIX Server, I end up doing overtime writing tools using shell scripting in Korn Shell environment. I was wondering if there is some way possible, so that I could setup similar environment at home using Linux Desktop, like Ubuntu, Fedora, etc and avoid doing overtime 🙂

AIX Server uses ksh as the default shell for every user account, whereas most of the Linux OS (I guess) have bash as the default shell. As AIX is proprietary server made by IBM, so I opted for Virtual Box instead, where I could do all my exploration.

Why Fedora 12?

Just download the ISO Image for suitable flavor of Linux OS available in the Open Source community. Thankfully, I had Fedora 12 ISO Image, which I downloaded back in 2010. To save time, I used the same to setup VM using Virtual Box. 

While installing the Fedora 12 VM (on Windows 7 Host OS), I created user account [nvarun], which by default has bash shell environment loaded whenever I open the Terminal desktop application to do command line operations.

Setting up Korn Shell

I have spent past few weeks, trying to figure out how to change default shell for user account and loading ksh environment, whenever I open Terminal desktop application and user account [nvarun] logs in by default.

  1. Open Terminal desktop application
  2. By default, the user account [nvarun] is logged in.
  3. Execute following command;
    $> grep nvarun /etc/passwd
    nvarun:x:500:500:Varun Nischal:/home/nvarun:/bin/bash
  4. As you would notice, the entry is colon-delimited and the last delimited field tells you the default shell for user account.
  5. Now to change the default shell to /bin/ksh, execute following command;
    $> sudo chsh -s /bin/ksh nvarun
    Changing shell for nvarun.
    Shell changed.
    $> exit
  6. As you exit from Terminal and re-open it again, you won’t see the change yet.

Before we proceed further, it is important to understand what AIX Server does with user accounts.

The shell uses two types of profile files when you log in to the operating system.

The shell evaluates the commands contained in the files and then runs the commands to set up your system environment.

The files have similar functions except that the /etc/profile file controls profile variables for all users on a system whereas the .profile file allows you to customize your own environment.

As we need to customize the our environment, in that case, use vi editor to edit the $HOME/.profile file and add following line;

export ENV=/etc/kshrc

The /etc/kshrc file is a shell script that customizes the Korn Shell environment and often contains list of environment variables, aliases and function definitions, etc.

Each time you start a new instance of the Korn shell, the ksh command examines the value of the ENV environment variable set in the $HOME/.profile file.

If the ENV environment variable contains the name of an existing, readable file, the ksh command runs this file as a script.

By convention, this file is named $HOME/.kshrc. You can use another name, but you must set the ENV environment variable to point to it.

Now, you may exit from Terminal and re-open it again, the changes would be reflected and this lets nvarun create aliases, functions, etc. to customize the environment further.

I am already excited to explore further, hope this helps 🙂