Article | Vectors, Strings and Transform Functions

Hardly, one would find blogs/articles/tutorials on Ab Initio transformation functions. I introduce you to another set of functions which will make you understands vectors in a different way.

Scenario

Suppose, there is data file whose record format has 5 fields, namely A, B, C and D. Let us give them useful meaning, A – student_id, B – student_nm, C – sem_num, D – subject_name, E – marks.

There will be a possibility that combination of A, B remains the same across several records in the data, whereas remaining columns may change. For instance, student Mr. X has 3 subjects in semester 1, whereas Mr. Y has 4 subjects in semester 2 and Mr. Z also having 4 subjects, but in semester 2.

What if there’s a need to consolidate so much information in a single record and present it as a report. This report must have Student Name, Student Id, Semesters covered so far (comma-delimited), Courses taken (pipe-delimited) and Average Marks.

Transformation Function

The following transformation rule uses string_join() and vector_sort_dedup_first() to solve the purpose, i.e. to get an output string of pipe-delimited vector elements.

out.target_field :: string_join(vector_sort_dedup_first(in), "|");

One can create user-defined function which wraps the above functionality, which would be generic enough to be used in any transformation where vectors are involved.

out :: get_delim_vec_ele(input_vec, delim)
begin
   out :: string_join(vector_sort_dedup_first(input_vec), delim);
end;

This function takes 2 arguments, input vector and delimiter to form delimited string of vector elements as output. So, your actual transformation can now look like;

out.target_field :: get_delim_vec_ele(in, "|");

Hope this helps :)

taT4Nix | convert EBCDIC to ASCII and vice versa

So far, I have encountered this on data warehousing projects, probably this might happen in some other domains too. Anyways, if there’s an ebcdic file with you, mostly retrieved from Mainframe systems. Then, one would like to convert them to ASCII for making modifications using text editors on UNIX servers, like AIX.

I have used the following command several times for changing the file from ASCII to EBCDIC or vice-versa. So, this is how its done;

dd if=<ebcdic-file> of=<ascii-file> conv=ascii

Now, you can start modifying the ASCII version and once done, you may convert it back to EBCDIC to be used by your application.

dd if=<ascii-file> of=<ebcdic-file> conv=ebcdic

If you’re just replacing particular number of bytes with equivalent number of bytes having different characters, the conversion would be smooth and application reading the file should not have any issues.

However, I had some issues while adding/deleting records into/from ASCII version, as found when converted back to EBCDIC mode. The file was unreadable by the application and I had difficulty reverting without having backup of EBCDIC version.

Hope this helps :)

AutoSys Jobs, ZQL and Oracle Database

Since few months, I have been dealing with AutoSys Jobs at workplace and so far, have been running sendevent commands for triggering the AutoSys Jobs to handle commonly used scenarios, covered as follows;

  • CHANGE_STATUS
  • KILLJOB
  • FORCE_STARTJOB
  • STARTJOB
  • JOB_ON_HOLD, JOB_OFF_HOLD
  • JOB_ON_ICE, JOB_OFF_ICE

However, time had come that I started modifying the AutoSys Job definition using JIL, i.e. Job Information Language. So, to gain better understanding I started referring Unicenter AutoSys Job Management Reference Guide and have been exploring ever since.

AutoSys Relational Database

The interesting thing to know, the AutoSys Jobs definition, runtime information, etc is stored in Oracle Database and the guide states the following;

Because Unicenter AutoSys JM uses a relational database, you can query the database to supply custom reports and information.

Note: Using SQL commands to change information in Unicenter AutoSys JM tables may cause your system to fail. The Unicenter AutoSys JM tables and views listed in the following sections are for documentation purposes only. They are not intended to be used as a public interface and are subject to change between releases.

By default, the Unicenter AutoSys JM OS user is “autosys”. However, you may define a different ID for the OS user during installation.

So, the query language used for querying purpose is known as ZQL.

ZQL

If your Unix server has AutoSys installed, chances are you can view man pages as well for ZQL. I will give a brief overview in this blog post, through following ZQL query I wrote recently;

SELECT
A.as_group||','||B.text||','||A.job_name
FROM
<database>.ujo_jobst A,
<database>.ujo_intcodes B
WHERE
A.job_type = '<job_type-attribute>'
AND
A.as_group = '<group-attribute>'
AND
A.status = B.code
ORDER BY
job_name,
text
;

You may click on the hyperlink in the above query to learn more about the table/view definitions.

The column ujo_jobst.as_group (pulled from ujo_job2 table) contains the group information, which is usually specified in defining the AutoSys Job as follows;

group: <group-name>

The column ujo_jobst.job_type (pulled from ujo_job table) is straight-forward, contains either b/c/f, depending on whether its a Box/Command/File Watcher AutoSys Job.

job_type: b

The table ujo_intcodes, contains 3 columns only and code column is equivalent to the ujob_status.status column. So, the ujo_intcodes.text column contains values corresponding to each code like RUNNING, ACTIVATED, INACTIVE, FAILURE, etc.

As a result, I have pulled a report by executing ZQL script, having group-specific box AutoSys jobs along with their current status ;)

taT4Nix | Redirect STDERR to File or File Descriptor

$> cd /
$> find $PWD -type -f

If you change directory to root directory on any UNIX server and try the above find command, chances are you would see long list of error messages along with few search results.

Mostly, the error would be Permission denied, one would like to suppress such messages, as they are not helping in achieving the desired results. In that scenario, do the following;

$> cd /
$> find $PWD -type -f 2> /dev/null

There you go, looks simple isn’t it?

2> means redirect STDERR to the file or file descriptor on the right-hand side of output-redirection operator.

In our case, it would be a file known as /dev/null, however there could be scenarios when your command’s output is redirected to some file, yet the error messages are getting printed on the STDOUT (as it is by default).

To handle that, do the following;

$> cd /
$> find $PWD -type -f > $HOME/sample_logfile.log 2>&1

Using 2>&1 would redirect the STDERR to file descriptor for STDOUT, i.e. &1 However, STDOUT is already redirected to $HOME/sample_logfile.log So, effectively STDERR and STDOUT are both being redirected to same logfile.

That’s all for today, keep scripting :)

Article | Automate Backup for IE Favorites

Before diving deep into the world of shell scripting, I had ventured with batch scripting on Windows XP machine. Well, recently I had to format Windows 7 running on Dell Studio laptop.

In doing so, I forgot to keep backup of the browser bookmarks/favorites, even though Chrome/Firefox provide options to export them. However, I had been using IE lately and kept rather few, yet important links bookmarked.So, I rarely cared about backing them up until recently.

Why not Google?

Well, I did few days back and found the following link useful;

http://support.microsoft.com/kb/211089

As most of the Windows users would be knowing the OS keeps your favorites/bookmarks under following directory structure;

Windows XP - C:\Documents and Settings\<username>\Favorites

Windows 7 – C:\Users\<username>\Favorites

Generally, Windows OS, be it XP or 7, would be installed in C drive, part of partitioned hard disk storage. So, I was wondering why not make use of the other partitions which are least impacted by crashing/formatting of the Operating System.

Also, keeping backup of the folder manually regularly would not be that easy, even though it is.

Why not automate the backup process?

This is what I think should be done (technically), by creating the following batch script;

:: Assuming the system has another partition as E drive

cd /d "E:"
rmdir /s /q Favorites
cd /d "C:\<YOUR-DIRECTORY-STRUCTURE>"
xcopy /s /f Favorites "E:\Favorites"

Now, I just have to save it as batch program, keep it on any drive other than the OS drive.

I tested it on Windows 7 and worked well.