Using NetStat to get process name from port number or IP addreess

NetStat is one of powerful Unix command line tools that is present on pretty much every machine, simple to use and yet is very powerful in the hands of advanced user. With NetStat you can debug network problems without installing any third-party software – you just need to learn a few command line parameters.

One command that I use very often is

netstat -tuapn

This command displays all connections (both incoming and outgoing) with corresponding IP addresses, port and process/PID numbers. Command parameters are

-t Show TCP connections
-u Show UDP connections
-a Show both listening and non-listening sockets.
-p Show the PID and name of the program to which each socket belongs
-n Show numerical addresses instead of trying to determine symbolic host, port or user names.

Output of this command is in the following format

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        1      0 2401:db00:11:2a:face::43936 2401:db00:11:2a:face:0:1456 CLOSE_WAIT  14971/hhvm
tcp        0      0 2401:db00:11:2a:face::37665 2401:db00:11:2a:face:0:9100 ESTABLISHED -
tcp        1      0 ::1:20845                   ::1:9099                    CLOSE_WAIT  14333/hhvm
udp        0      0 127.0.0.1:53                0.0.0.0:*                               -
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               -

You can easily find program responsible for specific connection by running following command (just replace 20413 with PID number you get from netstat output)

[sergey ~] ps -p 20413 -o comm,args=ARGS
COMMAND         ARGS
python          /home/sergeyma/Enthought/Canopy_64bit/System/bin/python -c from IPython.kernel.zmq.kernelapp

If you want to find all connections to specific IP address you can use

netstat -tuapn | grep 10.4.170.85

To find all connections pointing to specific port run

netstat -tuapn | grep :123

To find all connections from specific program first you need to get PID of process you are interested in (replace python with process name) and then run netstat

ps aux | grep python | grep -v grep
#Find PID of the process in the second column
netstat -tuapn | grep $PID #Replace 20413 with process number from previous step

Netstat is very versatile program that is very useful in multiple scenarios. I highly recommend to use it when debugging network issues or finding process responsible for specific port or communication with specific IP address.

Posted in linux, productivity, tools | Leave a comment

Copying to OSX Clipboard from Remote Linux Terminal in iTerm2

Many people know about extremely useful program pbcopy that copies from stdin to Mac clipboard. You can do something like cat urls.txt | pbcopy to copy content of urls.txt to Mac clipboard. Easy and very useful.

Unfortunately when you connect to Linux box using iTerm2 you could not just call pbcopy – there is no such command in Linux and besides you are using remote box. But there is a very nice solution if you use iTerm2. iTerm2 allows terminal program to control clipboard using escape codes. So below is bash function that is analog of pbcopy on Mac. To use it add three lines below to your .bashrc file.

function pbcopy {
    base64 -w 0| xargs -0 -d "\n" -n1 -I {} bash -c "echo -ne '\e]52;;{}\e\\'"
} 

Now if you want to copy output from some program to Mac clipboard you can just do

> cat data.txt | pbcopy 

One significant limitation is that you could not copy more than 500-600 bytes this way.

Posted in linux, mac, tools, unix | 1 Comment

Using GNU Parallel to speed up and simplify data analyzes

GNU Parallel is one of the lesser known tools for Unix that can significantly speed up data analyzes. Parallel is a tool to run commands in parallel in different processes. It has much easier to understand syntax than xargs and allows to do many other things that otherwise require actual programming. Below are some examples that illustrate how parallel can be used for data analyzes.

Fetching data from multiple domains

parallel wget ::: www.domain1.com/file1.zip www.domain1.com/file2.zip www.domain1.com/file3.zip 

This command will in parallel download files from respective URLs listed in command line. This is convenient if you want to download just a few files

If you have a list of files to download in a file urls.txt, you can use following command

parallel -a urls.txt wget

Alternatively you can use piping to supply a list of files

cat urls.txt | parallel wget

Do time consuming analyzes on multiple files

Let’s say that you have a list of files that you want to run some analyzes on. You have fancy 2 CPU 32 core machine but your algorithm is not easily parallelizable (or you are just too lazy to make it work in multithreaded mode). Sometimes you can save significant time by processing files at the same time in multiple processes.

ls | parallel processing -i {} -o {.}.out

This command with run ./processing on all files in current directory. {} – is input parameter, in our case file name. {.} is input parameter with stripped out extension. By using this command with we will put output information into filename.out.

You can also use Unix globbing to select files. Command below runs processing command on all files with *.dat extension in current directory.

parallel processing -i {} -o {.}.out ::: *.dat 

Split huge file and process smaller parts independently

If you have one very large file and you want to process every line, you can split the file into multiple smaller one using split command and then use parallel to process multiple files in parallel.

#split file into chunks of 5000 lines each. 
#output files will be called data_split_aa...split -l 5000 data data_split_ax
split -l 5000 data data_split_

#process all files from split
parallel processing {} ::: data_split-*

Trying out multiple parameters

You can use parallel to iterate over multiple parameters – no need to write nested loops!

[~/parallel]$ parallel processing {1} {2} {3} ::: A B C ::: 1 2 3 ::: X Y Z
A 1 Z
A 1 Y
...
C 3 Z

Command above runs processing on all combinations of {1, 2, 3}, {A, B, C} and {X, Y, Z}.

Use parallel to run multiple command at the same time

One of the easiest use cases for parallel is to run multiple different commands at the same time. Just put whatever you want to run in a single file and run parallel on it. It’s similar to cat file_to_run | sh but much faster.

parallel < files_with_commands_to_run.sh

GNU Parallel is not part of most Linux distributions but you can get it here. More detailed documentation is available at http://www.gnu.org/software/parallel/man.html .

Posted in data analyzes, linux, tools, unix | Leave a comment

Capturing Screen Recording on MacBook

Many people do not know that every Mac computer has pre-installed screen video capture utility. All you need to do is start QuickTime Player, go to File->New Screen Recording and start recording video.

Posted in mac | 1 Comment

Google Chrome Shortcuts

Many people do not know about excellent shortcuts that Google Chrome provides on both Microsoft Windows and MacBook. Below is list of shortcuts that I use daily

⌘-N Opens a new window.
⌘-T Opens a new tab.
⌘-Shift-N Opens a new window in incognito mode.
Press ⌘-Shift and click a link. Or press Shift and click a link with your middle mouse button (or mousewheel). Opens the link in a new tab and switches to the newly opened tab.
⌘-Shift-T Reopens the last tab you’ve closed. Google Chrome remembers the last 10 tabs you’ve closed.
⌘-W Closes the current tab or pop-up.
⌘-Shift-W Closes the current window.
⌘-L Highlights the URL.
⌘-Option-F Places a ‘?’ in the address bar. Type a search term after the question mark to perform a search using your default search engine.
⌘-R Reloads your current page.
⌘-F Opens the find bar.
⌘-D Saves your current webpage as a bookmark.
⌘-Shift-Option-V Pastes content without source formatting.
⌘-Option-I Opens Developer Tools.
⌘-+ Enlarges everything on the page.
⌘ and - Makes everything on the page smaller.
⌘-0 Returns everything on the page to normal size.

You can find full list of shortcuts for Mac at https://support.google.com/chrome/answer/165450?hl=en and for Windows at https://support.google.com/chrome/answer/157179?hl=en

Posted in mac, productivity, windows | Leave a comment

Using iTerm2 with Growl for Notifications

When I compile or do any other operation in terminal that takes a long time, I usually switch to browser or start doing something else – who wants to stare at terminal doing nothing for 10 minutes. Before I started using Growl, I was constantly switching forth and back from terminal to check if program finished compiling. It was distracting and unreliable. Sometimes there was error in code and compiling failed in less than a minute, but I did not know about it until I checked terminal five minutes later.

To fix this I use Growl – well-known notification program for MacOS. You have to pay for Growl but benefits outweight small price you have to pay for it (alternatively source code is available and you can build it yourself for free). I use MacBook Air as desktop with almost all work done on Linux server – so script is for Linux.

What I did before

[sergeyma@dev1456 ~] make

What I do now

[sergeyma@dev1456 ~] make; growl "make finished"

When make finish working, I get a nice notification from Growl – I know that operation ended.

Setting it all up

Add growl function to your .bashrc file in Linux

# A bash function to display a growl notification using iTerm's magic
# escape sequence. This version will work under screen.

growl() {
      local msg="\\e]9;\n\n${*}\\007"
      case $TERM in
        screen*)
          echo -ne '\eP'${msg}'\e\\' ;;
        *)
          echo -ne ${msg} ;;
      esac
      return
}

Enable iTerm2 integration in Growl

That’s it – I hope it will save you time!

Posted in linux, mac, productivity | 3 Comments

Moving and Resizing Windows on Mac

Microsoft Windows has excellent keyboard shortcuts that control window size and position and let you easily move window from one display to another (if you have two display setup). When I started using Mac I was surprised that by default Mac does not have any shortcuts that reliably control window sizes – to maximize app I had to use mouse to click on small green circle in the upper left corner of the window. It was extremely annoying especially because I use notebook and often switch between desktop setup (with 30″ external monitor) and tiny 13″ MacBook Air display.

Fortunately there is excellent app that solves this problem – Spectacle. It has all the features that I need – maximize window, move window from one side to another and move window to one of the corners. My productivity was immensely improved from using this small utility.

I did not like default settings for keyboard shortcuts in Spectacle, so I set shortcuts to the one below.

screenshot

Posted in mac, productivity | Tagged | Leave a comment

Transposing columns and rows in Unix/Linux

There is no direct command to transpose columns/rows but you can easily do it using command tr. This is often necessary when you want to find out total number of columns or ordinal number of some specific column

For example you want to number columns in a file

[sergeyma@dev1456 ~/scripts] cat f1
column1 column2 column3
23 12 12
3 23 32

Command below outputs first line of file using head and then transposes this line by replacing tab with end of line.

[sergeyma@dev1456 ~/scripts] head -n 1 | tr "\t" "\n"
column1
column2
column3

In this code, head outputs first line of file, tr replaces tab with end of line and then add line numbers in output using nl.

[sergeyma@ ~/scripts] head -n 1 f1 | tr "\t" "\n" | nl
1 column1
2 column2
3 column3
Posted in productivity | Leave a comment

New batch of photos from Tanzania

Getting to Machame Gates

http://www.sergeymarkov.com/blog/travel/tanzania/climbing-kilimanjaro-getting-to-machame-gates/

Posted in Uncategorized | Leave a comment

First photos from Tanzania

You can look at them here

Posted in Uncategorized | Leave a comment