Task Automation without API through Headless Browser

I've been researching lately on headless browsing as I have this automation idea that can literally put me on early retirement. For those of you who are looking to automate some task but the Site you're relying on don't have any available API to work with then this might solve your problem. Special thanks to my colleague Huy who suggested the idea.


Undertanding Headless Browser:
https://en.wikipedia.org/wiki/Headless_browser

PhantomJS - This is a good headless browser
http://phantomjs.org/

CasperJS - You can create simple javascript code using this which makes interaction and navigation with your headless browser so much easier.
http://casperjs.org/

Modify FSTAB on CentOS or RHEL 6 on Bootup

Ever encounter a problem when you can't boot due to corrupted or wrong FSTAB entry. Here's a simple fix

Problem:

Solution:

a. At  Grub press any key
b. Press "e" to edit

c. Under kernel parameter Append this command "init=/bin/bash"
d. Press Enter then "b"
e. This will take you directly to #.. Remount the / by typing  mount -o remount,rw /
f.  Edit fstab vi /etc/fstab
g. Save and Reset the system

Annoying "perl: warning: Setting locale failed"

This post shows a simple way of removing this annoying warning when running perl in linux.


perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = "UTF-8",
LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").

Solution:

on your home directory:


1. Edit your user bash_profile

vi .bash_profile

2. Insert the following:

export LANGUAGE=en_US.UTF-8

export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

3. Rerun your Bash Profile

. ~/.bash_profile 


Hope this helps.

FIND and REPLACE TEXT inside a file without using TEXT Editor

FIND and REPLACE TEXT inside a file without using TEXT Editor


You can perform a text replace within a file without using VI or other text editor available on your Unix or Linux box. Below command will help you perform that action. 


Using SED:
sed -i 's/FIND TEXT/REPLACE TEXT/g' FILENAME.TXT


Using PERL:
perl -p -i -e "s/FIND TEXT/REPLACE TEXT/g" FILENAME.TXT




Where:
FIND TEXT - Content you want to replace
REPLACE TEXT - New Value of the content
FILENAME.TXT - Filename

FOLDING IN LINUX

Fold Long Line files to Finite Width output device 
files.

Ex: fold -b < long_line_file >folded_file

Now the folded_file will  contain output with newlines 
inserted after maximum bytes specified.  This will be 
useful when editing special files. You can do the reverse 
using "expand" command.

BAD and BETTER COMMANDS

BAD and BETTER COMMANDS

I cringe anytime I see someone code inefficiently.  Here are
three of the most common mistakes, followed by a better way to
do the same thing.

Bad:    cat somefile | grep something
Better: grep something somefile
Why:    You're running one program (grep) instead of two (cat
and grep).

Bad:    ps -ef | grep something | grep -v grep
Better: ps -ef | grep [s]omething
Why:    You’re running two commands (grep) instead of three (ps
and two greps).

Bad:    cat /dev/null > somefile
Better: > somefile
Why:    You're running a command (cat) with I/O redirection,
instead of just redirection.

Although the bad way will have the same result, the good way is
far faster.  This may seem trivial, but the benefits will really
show when dealing with large files or loops.

I SED BLANK

I SED BLANK

Using sed, you can remove blank lines, and
lines that contain only whitespace, from a
file using the following:

sed -e '/^[     ]*$/d' InputFile >OutputFile

Within the single quotes ('), the forward
slashes (/) delimit the regular expression
that will be interpreted by sed.  The "d"
before the closing single quote, tells sed
to delete any lines that match the regular
expression.

Within the regular expression, the caret
(^) matches the beginning of a line.
The []* matches zero to many occurrences of
the character list between the open bracket
([) and the close bracket (]) (in the above
regular expression, you must insert a space
and a tab between the brackets).  The dollar
sign ($) matches the end of a line.

These three constructs together match any
blank line or any line that contains only
spaces and tabs (in any combination).

Since the standard operation of sed is to
echo lines to stdout, all lines except blank
lines (or lines that only contain whitespace)
will be sent to OutputFile.