Monday, January 31, 2011

How to use GREP command in Shell


Introduction

Sometimes you remember a phrase or a given word or words, you put in a document, but you do not remember the name of the document. How to find it?, well do a search of all your documents, looking for that word or words or sentence, in other words, look for a string or strings.
Now, thanks to computers this is easier now, than it was in our parents' days.
If you are using Linux, you have grep to help on this job.

Look for a document, containing a given string

First the easy case, you know the exact sentence, you are looking for, and you at least remember the folder where the file is.
grep "sentence to look for" /home/user/docs/
Now, let's suppose you do not know if the sentence was in uppercase or in lowercase, so ask grep to ignore case.
grep -i "sentence to look for" /home/user/docs/
Well, let's suppose you have a lot of sub-folders, and you do not remember where your file is.
grep -r "sentence to look for" /home/user/docs/
Let's see an example of the output in my PC
grep -ri "Introduction" /home/user/post/
The output is:
/home/user/post/monthy-newsletter.txt:###Introduction###
/home/user/post/monthy-newsletter.txt:I'm no expert in MySQL, but anyway I have written three introduction-type MySQL posts, something we all need to know to start [how to create databases][1], [How to list those databases][2] and [How to create tables in MySQL][3]
/home/user/post/how-to-debug-bash-shell-scripts.txt:##Introduction##
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt:##Introduction
/home/user/post/interview-raphael-hertzog.txt:I also have plans for bigger changes concerning Debian, and among them is the introduction of Debian Rolling, a distribution similar to testing but with some design choices to make it more usable at any point in time.
/home/user/post/four-years-with-debian-testing.txt:##Introduction##
Binary file /home/user/post/.find-a-documents-with-given-string.txt.swp matches
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt:##Introduction##
As you can see there are some binary files, also scanned, if we want to avoid that:
grep -riI "introduction" /home/user/post/
The output now is:
/home/user/post/monthy-newsletter.txt:###Introduction###
/home/user/post/monthy-newsletter.txt:I'm no expert in MySQL, but anyway I have written three introduction-type MySQL posts, something we all need to know to start [how to create databases][1], [How to list those databases][2] and [How to create tables in MySQL][3]
/home/user/post/how-to-debug-bash-shell-scripts.txt:##Introduction##
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt:##Introduction
/home/user/post/interview-raphael-hertzog.txt:I also have plans for bigger changes concerning Debian, and among them is the introduction of Debian Rolling, a distribution similar to testing but with some design choices to make it more usable at any point in time.
/home/user/post/four-years-with-debian-testing.txt:##Introduction##
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt:##Introduction##
Finally, I just want the file names, and not the sentences where the sentence or word appears.
grep -riIl "introduction" /home/user/post/
The output will be:
/home/user/post/monthy-newsletter.txt
/home/user/post/how-to-debug-bash-shell-scripts.txt
/home/user/post/how-to-setup-dns-bind-master-slave-linux.txt
/home/user/post/interview-raphael-hertzog.txt
/home/user/post/four-years-with-debian-testing.txt
/home/user/post/how-to-change-the-priority-of-Linux-processes.txt

Conclusion

I know, you may use other ways, maybe find, but I like this method, and use it a lot, just like the advanced search function in Gmail :

15 Days To GATE: How To Prepare Now


Last fifteen days left for the exam that will decide what and how your life will be. If one prepares well have to look for M.Tech admissions and worry about cutoff scores. Even now, if one is having fun with friends and wasting time on movies and drama serials, must start looking for things to do after their engineering.
Fifteen days’ too less time and sincere students preparing for GATE must be confused with so much read. What you have learnt in last four years have to be brushed up in just fifteen days. Looks difficult but is for sure possible.
In this post present the strategy to get ready for GATE. Remember these and try to adopt, they will definitely help you to grow.
Avoid going out: Student who are staying hostels are lucky and can easily think of not going out of their room except for food, (meals three time and some snacks). This will save your lot of time and make you think why are you at your hostel room? And I hope that will eventually help to concentrate more.
Don’t waste time reading all the concepts again: This is not the time to start reading again from scratch. You have to go ahead with short note and revise whatever you have read and practice questions and answers.
Solve papers and Question Bank Questions: Try to solve as many papers as you can if you are not able to solve or get correct answers, go back to notes or explanations and understand that from book and move on. Don’t go back to texts and stay their all the day long.
Easy Subjects first: I read a quote few days back, “If you have to eat four frogs, eat the biggest first.” This was about facing challenges and if you have too many difficult things you have to start with most difficult one by which you can relax a bit at end. But my idea is different specially when time is less and you know you can run short of time and miss few of them. Its better to start with the subjects you are confident in and you have already read. Make sure that you get everything there. Solve its questions and learn the concepts well. Stay confident about them and that will help you with other subjects and built your confidence.
Don’t think too much, start working: Another quotation “Leaders start with the end in mind”. I would say that don’t see the end start working. Without giving a tough time you cannot get anywhere. If you think about all subjects you need to finish, you will feel totally burdened. Just plan out what you want to study today and be sincere and complete it today. You know the end, that is cracking GATE, you all need to do is to be sincere and keep working for it.
These 15 days are not yours: Back in 1997 I had my 10+2 (pre university or intermediate) exams one of my friend’s family decided to watch DDLJ (Dil Wale Dulanhiya Le Jainge) and rented a VCD player along with a color TV. Almost all the street children were at their home. I too went there, sitting on the floor and desperately wanted to watch the movie. As soon the titles started I thought I have exams in few days, if I study now I can watch it later. I get only a chance to do great in exams but there can be many chances to watch this movie latter too. So I just moved on with all my friends enjoyed and discussed it for days. That day made me strongest in my will and desires and actions. Since then I watch cricket matches, news and read news only when I am free. I have good things to do than getting attracted and affected with all the drama going around. With fifteen days left for GATE exam I ask you to ask yourself what do you want in life and what things can you do after GATE exam.
With less time left for GATE exam I call you come to life. Decide what you want in life and live the way you want it. Don’t let anything takeover your desire to be great. You can not be an example for you younger’s by watching movie and dramas all the day and night, chatting and roaming around markets. We are average students; we need to an extra effort to reach to higher level. The extra effort cannot come with wasting time.
I wish you all a great luck. Stay focused and work hard. Every single second is valuable and need to be used properly.

Wednesday, January 26, 2011

Happy Republic Day

Dear All,

WISH YOU A VERY HAPPY REPUBLIC DAY

JAY BHARAT

Monday, January 24, 2011

How to enable IP forwarding in Linux


Introduction

If you want to use a Linux Operating System driven box, to act as a router or gateway, you need IP packets to pass through your Linux box. By default packets not directly addressed to a Linux powered PC will be discarded, so you need to enable IP Forwarding.
This can be done in different ways, and we’ll check here some of the most common.

Use procfs

This is maybe the most used way, it is a temporary change, and you need to enable it after every reboot.
echo 1 > /proc/sys/net/ipv4/ip_forward
You can add this line to /etc/rc.local file, and that way, each time you reboot your computer it will be enabled again.
You can check if IP forwarding is enabled or disabled by checking the content of/proc/sys/net/ipv4/ip_forward file
cat /proc/sys/net/ipv4/ip_forward
If the output is 1, it is enabled if 0, then it is disabled.

Use sysctl

sysctl let’s you change Kernel values on the fly, so you can use it, to change the IP forward behaviour of your Linux.
First, let’s check if it is enabled or disabled, as root run:
sysctl -a | grep net.ipv4.ip_forward
Now you can set its value to 1, to enable ip forwarding.
sysctl -w net.ipv4.ip_forward=1
This is also temporary, if you want it to be permanent, you can edit the file /etc/sysctl.conf
And modify or add this line:
net.ipv4.ip_forward = 1
Now let Linux load the changes you’ve made.
sysctl -p
As root.
You’re done, now you have IP forward enabled on your Linux, you can now proceed to configure your Linux router or Linux gateway.
This works for Ubuntu, Debian, Fedora, Slackware or any other Linux distribution

Classifying Linux Distributions


Introduction

This is somehow my personal classification, of Linux distributions. And maybe at the same time of the Linux users.
I’m going to classify only those I have used more than just a few hours in a virtual machine.
[Update: 01/23/2011]. I’m adding a new section, Special Mentions with some distributions suggested by the readers, but not used by me

Linux for the newbies

This are distributions for newbies, or Linux newcomers as well as for users who do not have too much time to tweak and configure their PCs.
  • Ubuntu Maybe the most famous Linux distribution, and usually the first one any newcomer sees.
  • Mint Linux Mint, gained a lot of popularity, it is based on Ubuntu, but it comes with support for mp3 and flash and other stuff Ubuntu does not support by default. I really do not understand why it isn’t more famous than Ubuntu. Recently they released LMDE (Linux Mint Debian Edition) a rolling release based on Debian, I think this will give Mint more exposure.
  • PCLinuxOS This is another newbie oriented Linux distribution, more or less like Mint, it comes out of the box, with support for a lot of hardware, and plugins and coders for different media, so out of the box, you can start using it for almost any Desktop application.
  • Mandriva I forgot to mention this one, but I am updating the post once again.
    Mandriva is really a great distribution, I used it for over a year, and in that time, when I was using it, it was the most advanced Linux Desktop I have ever tested. It had support for a lot hardware other distributions did not at that time.
    Also they put a lot of focus on the eye candy, therefore, it is a great looking Linux distribution, it is aimed to be a newbie friendly. And it is that indeed.
    From Wikipedia:
    Mandriva Linux (formerly Mandrakelinux or Mandrake Linux) is a French Linux distribution distributed by Mandriva (formerly Mandrakesoft). It uses the RPM Package Manager. The product lifetime of Mandriva Linux releases is 18 months for base updates (Linux, system software, etc.) and 12 months for desktop updates (window managers, desktop environments, web browsers, etc.). Server products receive full updates for at least 24 months after their release.
    On September 18, 2010, due to financial uncertainty and the layoff by Edge-IT, a Mandriva subsidiary employing many of the corporate staff working on the Mandriva distribution, a fork of Mandriva named Mageia was announced
    I have not tested Mageia yet, if someone has, please let us know about it in the comments section

Intermediate Users

  • Fedora This distribution, is somehow the sandbox of RedHat guys, it is Desktop oriented, and usually packs bleeding edge software. They also make a lot of contribution to the Linux community. It use to be my main distribution once, but I do not like yum too much.
    I initially put this in the newbies section, but some readers (read comments below) convinced me to change its cathegory.
  • Debian Linux Well, I really do not know where Debian should be, to me personally, it fits on all sections, if you install it from the DVD, or from some of its Live Debian CDs, you may have something like Ubuntu, or you can use the netinst and end up with something like Gentoo or Arch Linux, and from there fully customize your Debian.

Advanced users

The distributions on this sections are not aimed for the newcomer, but you certainly does not need to be a Linux geek in order to install and use them.
  • Slackware Well maybe some slackers will say Slackware is not for advanced users, and that is partially true, if you install it, out of the box, it has almost anything you need to work, in a Server or Desktop environment, and if what it comes with, is all you need then you are OK, but if you want to install more software, you may be in troubles, if you have not use some command line tools before. I must say that Slackware is one of my favorite distributions, even though I’m really new to it.
  • Gentoo Now we are really talking about a distribution for advanced users, I’m using Linux for 5 to 6 years in a row now, and when I started with Gentoo, I noticed I know nothing about Linux. Finally I decided to stop using it, it is great, you can tweak almost, well not almost, just everything in your configuration. The problem is that with so much power at your fingers, you may screw things up easily. The community is probable one of the best, and the documentation is definitely the best in the Linux world. If you really have the time and you like Linux a lot, you should be running Gentoo.
  • Arch Linux This is probably my favorite distribution, I have not decided yet if it is this one, or Slackware or Debian. Let’s say the three of them are. Well talking about Arch Linux, this distribution is based on the KISS principle. It also used the BSD style init scripts, which to me are a lot easier to use than the System V ones.

Linux for Servers

Let’s list here some of the better distributions for servers, once again according to me.
  • CentOS If you do not mind having old software CentOS is a good server distribution, you can use CentOSPlus, but then you might be loosing one of CentOS strengths which is stability and security.
  • Slackware I have never used it as Server but the more I use it, the more I notice its stability, and Slackware actually focus on hard rock security and stability. Also very simple and easy to admin, it is great for the Server field.
  • Debian Well maybe the best Server distribution, as I said before, I do not like the system V init scripts. Also I do not like too much how it tweaks the configurations files, which are not standards but it is a great distribution to use as server. It has a lot of applications, maybe you will not be able to find one server application that is not in its official repositories. Some time ago, Debian backports became officially part of the Debian project, and now are supported by the Debian team. This just gives Debian more power as a Server distribution.
I know a lot of people use Ubuntu as server and Ubuntu has its Long Term Support (LTS) release, aimed to be used as server, but I have never used, I think “if I have Debian, why use Ubuntu?”.
I must say, that I have also used Arch Linux as server, and taking care when you upgrade it, it works great!. I like a lot the possibility to have the last versions of software available, without the need to compile by hand. I do not recommend anyway this for newbies or if you do not have the option to full backup your server before making any change to it.
I make a full backup make a change, and if something goes wrong I can go immediately back to the previous stage.

Special Mentions

[Update: 01/23/2011]
A lot of readers are mentioning OpenSuse, so I decided to put it here in a special section, I may add some others here, which I have not used, but according to the others the deserve a place here
  • OpenSuse From Wikipedia:
    openSUSE (pronounced /ˌoʊpənˈsuːzə/) is a general purpose operating system built on top of the Linux kernel, developed by the community-supported openSUSE Project and sponsored by Novell. After acquiring SUSE Linux in January 2004,[3] Novell decided to release the SUSE Linux Professional product as a 100% open source project, involving the community in the development process
  • This may be classified in both, newbies and Intermediate users, as it seems (according to its users) to be very user friendly

Conclusion

I know that writing this kind of personal articles, I can make a lot of people mad at me, please do not get mad at me, this is just my personal point of view.
But, at the same time, express your own thoughts and what you think about this topic and this Linux distributions, and feel free to list any one I have not listed, and let the rest know why you use it and what for.

Sunday, January 9, 2011

String and pipe command together in Linux shell

Introduction

When you are working in the Linux command line, you often need to type several commands in a succession, you can type one, wait until it finishes and then type the next one, or you can type all of them in a line, and they will execute in order.
There are various ways to execute commands in a line, I mean one after the other, we’ll see some of them in this article, so you can pick the best for your needs depending on the situation.

String commands in Linux using semi colon

This is the simplest way, for example:
 
htop ; free ; df -h

What you will get is htop executed, once you leave it, free will execute and output its info to the screen, immediately you will see the output of df -h on your screen.
It does not matter if the command is successful or not. So if you made a mistake and type this:
 
fre ; df -h

The output will be something like this:
 
bash: fre: command not found
Filesystem            Size  Used Avail Use% Mounted on
udev                   10M  212K  9.8M   3% /dev
/dev/disk/by-uuid/fc8f5296-7bd8-463a-87af-87c6d2215ff9
                       41G   23G   16G  59% /
none                 1004M  102M  903M  11% /dev/shm

You get the command not found error and then df -h is executed, so if you need that the first command be successful to execute the second or third one, do not use this method.

String commands in Linux using double ampersand (&&)

This is maybe one of the most known ways to string commands together in Linux, it works more or less the same as with the semi-colon, but it will check for errors in each command on the line before going to the next one.
This is great if you are compiling software to install it. For example:
 
./configure && make && make install

This way if any of the commands configure or make gives any error, the next one in the line will not be executed and all the execution is stopped.

String commands in Linux using double pipe (||)

With the double pipe “||” you are telling the shell that it should only execute the next command if the first one has errors. You can use it, if you want to perform an action and there are two ways to do it, so, you can join both of them with double pipes, and if the first one fails the second one will be executed.
Just as an example.
 
vmstat || free

So, only in the event you do not have vmstat installed (do not really know if that is possible) free will be executed, and you will have one way or the other the memory info your Linux box.
I’m sure you will find better applications :).

Pipe commands together with pipe (|)

Using a single pipe “|” to string Linux commands together will make another interesting thing, it will forward the output of one command to the other, the next command in the line, will use that output as its input.
I use this a lot when writing post for my blog, I write my post using vim and with markdown syntax, then I use the perl program to translate it to html, but also use smartypants, so I use a line like this:
 
/html-tools/markdown.pl mypost.txt | /html-tools/smartypants.pl | xclip -selection clipboard

That way, markdown.pl translate mypost.txt to html, then smartypants.pl adds the typographic punctuation HTML entities, and finally its output goes to my clipboard using xclip which I can just paste in my drupal blog.

Final notes

As you can see pipe commands in Linux is easy, and you have one option for each need, hope you find the one for your needs here.
If you have more examples or better ways to achieve these task, please share in the comments.
  

Stop or suspend a running process in Linux

Introduction

If you start a process on a terminal, being it local or a remote (ssh) terminal, you normally loose the control of the terminal while the process is running, then if you want to recover that control, you have some options.

Cancel the process

By pressing CTRL+C you can stop a running process, and recover the control of your terminal, this will tell to the running application to stop anything it is doing at that moment and shut down.

Suspend a process

Another way, is to suspend the process for a moment, you will use CTRL+Z for this, then you can resume the job in any moment, when you feel like to.

Restore your stopped processes

You have now two different ways to re-start your stopped processes, you can send them to the foreground or to the background, to send it to foreground execute:
 
fg n

To send them to background execute:
 
bg n

Where n is the number of the stopped process, and you will see it in brackets when stop it.
Like this:
 
[admin@arch ~]$ amsn                                                                                                                         
^Z                                                                                                                                             
[1]+  Stopped                 amsn

You can see that the stopped process is amsn and its identifying number is “1”. Now, if you ever forget the number, you can run:
jobs
you will get an output like this one:
 
[admin@arch ~]$ jobs
[1]+  Stopped                 amsn

How to change the priority of a Linux process

Introduction

The Linux Kernel will take care of the time each process, will have to use the processor, but you can change that, by changing the priority.
When is this needed?, well usually never, but if you have a not to fast PC, and you are compiling something, and at the same time having a video conferencing, you may want to increase the priority of your video conference applications, while reducing the priority of the program compiling your software.

How to increase or decrease priority

We will use nice in order to change the priority of a given process, first you need to know the PID of the program you want to re-prioritize. You can use pstree or htop to discover the PID, if you use pstree use it with the -p option.
 
pstree -p

Once you know the PID of the process you want to change its priority, use renice to change it.
The permitted range is from “-20” to “20” being “-20” the highest priority.
As root run:
 
renice -8 -p 123

To assign a priority of -8 to a process with a PID of 123
Remember that only root can increase the priority, and normal users, can only alter the priority owned by them, and only to decrease its priority, and not to increase it.
Read the man page of renice: also keep in mind that you may need to change the priority of a parent process to affect all children processes. As an example, I usually have 5 to 6 firefox processes, all nested to a parent process, so, work with the parent. Use pstree to find the parent.