Conditional parameters substitution in Unix

As a developer who is coding for future, there are conditions when we encounter uncertainty (Monads !! ) 

Shell scripts provide a way to construct some expressions based on these uncertainties. One way is to use “Dollar-Braces”

syntax for that is ${variable-defaultValue}

so if you start a shell and do 

echo ${myVar-variable not defined}

you would get response as

$> variable not defined

now if you do

myVar=nowDefined

echo ${myVar-variable not defined}

you will get

nowDefined

Advertisements

Linux Dollars($)

Linux/Unix shell provides some of very nice functionalities (That should always be kept in mind while developing own software).  While writing shell scripts there are number of inbuilt “special” variables provided for developers. Following are the variables that starts with dollar sign ‘$’

1. $# –> # sign occasionally stands for number and this variable tells number of positional parameters in a shell script eg. myCommand -u -v hello dev

has $# = 4

2. $- –> This one tells you about starting parameters for your shell (bash, korn etc)

if I run it on my bash terminal it gives output of himBH

3. $? –> ? is used to ask questions, here too it’s purpose is to ask: “what happened to last executed command”

0 == success

1 == failure

2 == some diaster

4. $$ –> What is my own PID

5. $! –> bang(!) pattern on shell has a use similar to its use here (for last command) here it stores the answer for “what is the process number of last background command”

6. $0 –> What is my name (shell script’s name) on the terminal it would give you -bash or -korn etc

7. $* –> list of all the shell argument

8 $@ –> Same as $* except when enclosed in double quotes

Monte-Carlo Method With Haskell

Going to use a package by Patrick O Perry.

The obvious problem that I face is that there is no documentation. Hope this blog save someone else’s time1. Installation (On linux for windows use cygwin ) [Tested on Linux  Mint and Windows XP

a. Install gsl-random.. Prerequisites is to install gsl for linux.. Follow the steps here

i) Go here –. http://mirrors.ispros.com.bd/gnu/gsl/

ii) Get the latest gsl-version.tar.gz from there. My case it was gsl-1.16.tar.gz

iii) Open the terminal and do the following

cd ~/Downloads

tar -xzf gsl-1.16.tar.gz

cd gsl-1.16/
sudo ./configure

sudo make

sudo make install

b.) Install gsl-random from hackage

1. sudo cabal install gsl-random

2. sudo cabal install monte-carlo

——————————————————Using the package———————————————————

1. Generate random number for various distributions (check)

———- Please note that all of the example is not 100 % my effort —–

—– in fact I have written only bmt and display random functions here —

<span style="font-family: ‘Helvetica Neue’, Helvetica, Arial, ‘Nimbus Sans L’, sans-serif; font-size: 15px; font-style: inherit; line-height: 1.625;">import System.Random</span></pre>
import Data.List
— Number of samples to take
count = 10000

— Function to process our random sequence
process :: [(Double, Double)] -> (Int, Int)
process = foldl’ sumInCircle (0, 0)

— Function to process a running value and a random value, producing a new running value.
sumInCircle :: (Int, Int) -> (Double, Double) -> (Int, Int)
sumInCircle (ins, total) (x, y) = (ins + if x*x + y*y [Double]
bmt (u1:u2:xs) = [sqrt(-2 * log u1) * cos (2 * pi * u1),sqrt(-2 * log u1) * cos ( 2* pi * u2) ] ++ (bmt xs)

— Function to display the running value.
display:: (Int, Int) -> String
display (heads, coins) = "pi = " ++ (show $ 4.0 * fromIntegral heads / fromIntegral coins)

— to display random numbers
displayRand:: [Double] -> String
displayRand [] = ""
displayRand (x:xs) = "list = " ++ show x ++ displayRand xs

— function to prepare the random sequence for processing
prep :: [Double] -> [(Double, Double)]
prep (a:b:r) = (a,b):prep r

main = do
g <- newStdGen
putStrLn . display .process . take count . prep $ randoms g
putStrLn . displayRand . take 100 $ (randomRs (0,1) g)
putStrLn . displayRand . take 100 $ (bmt (randomRs (0,1) g))
<pre>

 

API –> (should be similar to this one )

simulate 1000 (with randnormal 1000) (fun “callprice”)

—- plot random numbers generated by last and second last line —-

so the good news is that plot looks like they the random numbers are good enough to be used

— To Do —

— use MC simulations to get put and call price and compare with the Black scholes prices
—– To be continued ———

Web-Crawler (Python) for getting financial data

*** I have moved this blog to www.tanmaydutta.com

1. Pick website: [I am going to scrap http://finance.yahoo.com/q?s=DAIMX This is a random choice for an Indian equity and I am going to study a couple of more equities.
2. Create a folder to start your project –>

 scrapy startproject yahooFinance 

This should create the following kind of structure
folderTree
We will leave this structure as it is for now.
Next we just need to define some items to scrap from yahoo finance in items.py (which is located in yahooFinance –> yahooFinance –> items.py)
Now, we will write some basic code in items.py (This defines name and allowed domain)

#spider to crawl yahoofinance DMS India

from scrapy.spider import BaseSpider

class DMSSpider(BaseSpider):
name = "DMSE"
allowed_domains = ["finance.yahoo.com"]
start_urls = ["http://finance.yahoo.com/q?s=DAIMX"]

def parse(self, response):
filename = response.url.split("/")[-2]
open(filename,'wb').write(response.body)

Next go to spider directory
–> create a new spider which would crawl the website

#spider to crawl yahoofinance DMS India
#written by Tanmay
# Does not do any fancy thing.

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from yahooFinance.items import TickItem

class DMSSpider(BaseSpider):
name = "DMS"
allowed_domains = ["finance.yahoo.com"]
start_urls = ["http://finance.yahoo.com/q?s=DAIMX"]

def parse(self, response):
print 'in part'
hxs = HtmlXPathSelector(response)
sites = hxs.select('//span[contains(@id,"l10")]')
for site in sites:
item = TickItem()
item['name'] = [u'DMSTick']
item['value'] = site.select('text()').extract()
return item

Go to main project directory and start crawler by typing

scrapy crawl DMS

Two things (that I Can think of right now) can cause error at this point are :

#1 . Unknown command: crawl Use “scrapy” to see available commands More commands are available in project mode means you are not in the correct directory.

You have to be in a directory where your scrapy.cfg file is

#2. No crawler name “xyz” is found.

Make sure that the spider file that you just created has the “name” property exactly xyz or whatever you actually wanted

——————————————————-

Ok so if everything runs perfectly till now then that means basic functionality is completed.

—————————————————–

Next time will start beefing up the items.py to extract real data

# Define here the models for your scraped items
#
# See documentation in:
# http://doc.scrapy.org/topics/items.html

from scrapy.item import Item, Field

class TickItem(Item):
    # define the fields for your item here like:
    name = Field()
    value = Field()

Run it and save resuls in a JSON (DMSTick) by

scrapy crawl DMS -o DMSTick.json -t json

The final Json that this code produce is :

[[[{"name": ["demo"], "value": ["8.72"]}]
[{"name": ["demo"], "value": ["8.72"]}]

Hacking Emacs to work behind (maybe) productive office Environment

Problem 1. Your office has a proxy.

Solutions:


;----- EMACS BEHIND PROXY SOLUTION------------
 (setq url-proxy-services '(("no_proxy" . "work\\.com")
 ("http" . "put proxy here")))

;------------- END SOLUTION 1. ---------------------------------

 

 

Extra for emacs repos:


;--------------------- VARIOUS REPOSITORIES FOR INSTALL.EL-------
 (require 'package)
 (add-to-list 'package-archives '("org" . "http://orgmode.org/elpa/") t)
 (require 'package)
 (add-to-list 'package-archives
 '("marmalade" . "http://marmalade-repo.org/packages/"))
 (when (>= emacs-major-version 24)
 (require 'package)
 (package-initialize)
 (add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/") t)
 )

;------------------------ END REPOS----------------------------------

2. Problem 2
Your Office does not allow sourceforge for (maybe) productivity:

Get exuberantTags from github and build }:)

(You might need cygwin or mingw or vs) ( I have vs2008 and used nmake -f  [Name of the visuals_studo.mak]

Hopefully this one does add something to productivity.

 

Basic Development Enviornment setting on Linux Mint-15

1. For Nvidia (I have GT640 card)
Do not install the current-stable version but go for the latest [I installed nvidia-settings-310-updates]
2. For Emacs
The way I prefer (to facilitate organized dev structure as well):

1. Create two folders dev and appUtils (I have the same in my office dev pc)

2. Keep all the external “modes” into the appUtils->EmacsExternal folder

First things sudo apt-get install emacs-goodies.el

Second thing – setup install-package.el (Very important for later uses )

http://repo.or.cz/w/emacs.git/blob_plain/HEAD:/lisp/emacs-lisp/package.el

Third thing change theme (Otherwise I don’t feel like working 😦 )

3.  For development:

wxPython: Follow this link : http://wiki.wxpython.org/InstallingOnUbuntuOrDebian

in place of package [This might change when we have the actual package for ubuntu 13.04 with us]

    # wxWidgets/wxPython repository at apt.wxwidgets.org
    deb http://apt.wxwidgets.org/ natty-wx main
    deb-src http://apt.wxwidgets.org/ natty-wx main

Get git(Svn whatever you like I like git)

git:

1. cd dev

2. git init

3. sudo apt-get install xclip && ssh-keygen && xclip -sel clip < ~/.ssh/id_rsa.pub

4. git clone xyz@github/orwhatever-name-of-repo

Add Aliases and other time saving commands to the .bashrc
———————bashrc starts here——————————-

export EDITOR=emacs
alias l='ls'
# long listing of ls
alias ll='ls -l'
# colors and file types
alias lf='ls -CF'
# sort by filename extension
alias lx='ls -lXB'
# sort by size
alias lk='ls -lSr'
# show hidden files
alias la='ls -A'
# sort by date
alias lt='ls -ltr'
# replace history with h
alias h='history'
# ignore common commands
export HISTIGNORE=":pwd:id:uptime:resize:ls:clear:history:"
# ignore duplicate entries
export HISTCONTROL=ignoredups #args are ignoredups, ignorespace or ignoreboth
# control the size of the history file
export HISTSIZE=100000
export HISTFILESIZE=409600

Overzelous lamba usage

Was looking through a video on programming languages and realized I am also overusing lambdas sometimes when they are obsolutely not required. (Why you want to differentiate? -> because that’s the building block for writing elegant and clean scripts)

example of my own codes (From certain problems)

foldl (\x acc -&gt; x + acc) acc alist
--while this should be definitely written as
foldl (+) acc alist
--and moreover if you are just using some standard list then better to use
sum alist