Personal tools
You are here: Home kaeru's blog Unix

Unix

2007-12-28

Folding Shell Style Comments with Vim

Filed Under:

I often experience with people new to Unix or even people who have been using it for a while, that they tend to brush off vi/emacs editors quickly and then use simpler editors like pico, nano, gedit and the like.

They then never see why a lot of old time Unix system administrators and developers use these tools. vi for example was designed to edit files over a very slow connection (300bps). It's still very useful now, as lag still exists. It's very painful for me to see young developers use the mouse and scroll up, scroll down, when without lifting your hands from the keyboard you can jump to different functions, mark locations and jump back.

If you need to comment out some lines over a laggy ssh connection you can simple type :4,12s/^/#/ instead of type # and using cursor keys over the next 8 lines.

Speaking of comments, here's a common scenario. Sometimes configuration files are huge, because they're well commented with examples. You want to edit it, but sometimes you just want to see your modifications and only comments for the section you're looking at and not scroll through everything. You can hide the # comments with vim's folding feature.

So let's define a function to do this in our ~/.vimrc:

function! FoldShellComments()
    let &foldexpr = 'getline(v:lnum)[0]=="#"'
    g/.*/ if foldlevel(line(".")) > 0 | s/$/ !!!/ | endif
    set foldmethod=expr
endfunction

We then want to call this up easily so we define a user command:

command! -nargs=0 FoldShellComments :call FoldShellComments()

With this whenever we want to fold commented lines, we simple type :FoldShellComments This will tab complete by the way, you just need to type :Fo<tab>

That huge squid.conf file now looks like this:

http://kaeru.inigo-tech.com/blog/blog-images/squid.conf.png


2007-12-25

Reducing complexity and resource usage for Zope front-end

Filed Under:

Keeping things simple is important. Keeping things simple however does not necessarily mean things are dumbed down. There is a Unix mantra, which is to do one thing and to do it well. For example take bzip2, does it's job well which is to compress things, it doesn't do anything else. Yes it has a lot of options, but they're related to compressing data.

Sometimes when programs do too many things, they end up being bloated and complicated to set up. So when archiving http://www.apdip.net to a virtual server, it was good opportunity to simplify things, reduce resource usage, but not reduce functionality or performance.

So I've reduced the backend setup to a chain of:

squid -> pound -> zope
               -> cherokee 
  • Squid here does what it does best, which is to cache requests
  • Pound to load balance between application servers and httpd server
  • Cherokee a lightweight httpd server

Except for Squid, the others do their job really well with simple small configuration files. With squid, by keeping it's role to strictly as a caching server, the configuration is also simplified and less prone to errors.

Proper articles later, but hopefully some tips here will help you on your way.

Squid

Start here: http://wiki.squid-cache.org/Squid_Faq/ReverseProxy

Set our Squid to listen to port 80 and also deal with named vhost requests

http_port 80 accel vhost

Set Squid to go to Pound to manage rediretions and load balance the backend services

cache_peer 127.0.0.1 parent 81 0 originserver default

We needed to deal logs for different vhosts, and that was not too difficult. First we set the ACLs

acl apdip dstdomain www.apdip.net
http_access allow apdip
acl stats dstdomain stats.apdip.net
http_access allow stats

Then we split the logs by referring to the acls, so that each site has their own logs

access_log /var/log/httpd/www.apdip.net/access.log combined apdip
access_log /var/log/httpd/stats.apdip.net/access.log combined stats

Pound

We then configure Pound to deal with the backend services. Just man pound, the man page is all you need to setup different priorities of servers, redirect requests to different servers, set time outs on backend pools etc. It's quite simple, because well.. that's what Pound is supposed to do, and do well.

Here it's listening on 81, and redirecting the right requests to the right servers.

User  "www"
Group "www"

Client 300

ListenHTTP
    Address 127.0.0.1
    Port 81
End

Service

    HeadRequire "Host:.*stats.apdip.net.*"

    Backend
        TimeOut 120
        Address 127.0.0.1
        Port 8081
    End
End

Service
    Backend
        TimeOut 120
        Address 127.0.0.1
        Port 8080
    End
End

Cherokee

http://www.cherokee-project.com/themes/cherokee/indiankid.png

Love it.. small and simple configuration files. vhosts, cgi etc. snap to set up. The configs are even set up debian style (sites-enabled), brilliant. It's also uses much less memory than Apache and since we don't use any of the modules, php support etc, it makes much more sense. I got introduced to this by Alvaro some time back, since he wrote it, I trust it's as good as he says it is. :)

I won't paste the config files here, you can check them out, but all I can say is that it was simple, just go do the documention on the website to see examples for common uses.

2007-12-21

apdip.net on gambit

Filed Under:

Just moved apdip.net from servers in Bangkok to Inigo's server in Kuala Lumpur, remotely of course.

These kind of tasks are quite easy with standard unix tools such as ssh and tar. Assuming of course you're familiar with them (hint LPI Level 1 objectives are important). Setting it up was made even easier thought the use of FreeBSD jails, which allows me to host it on a virtual server which shares some of the host directories, but has it's own ip, own configuration and own third party (ports) software installed. This method shares kernel also, so it's very efficient.  This allows me to make very efficient use of my resources. As an SME it's very important to keep costs low, and if servers are able to do more, then it's better for the bottom line.

Unlike proprietary software, there are no silly things like registration keys, shared registry tied to physical server and other mechanisms that make it difficult to copy and move things. Every single software I need to duplicate it is available for free without any feature restrictions. Furthermore software versions are also not usually tied to specific OS releases. Meaning that I can run python 2.3 and zope 2.7 on FreeBSD 6.2 while the older server was on FreeBSD 5.3. I could replicate it on different Linux distributions too. All these kind of freedoms for the user, end up making things look amazingly easy to do compared to doing it on a platform such as Microsoft. Latest Plone 3.0 will run on FreeBSD 5,6 and upcoming 7, Fedora, Centos, Suse and even Windows.  Contrast this to Sharepoint 2007 which requires Windows 2003. Running Windows 2000? Tough.. pay for more licenses. Already have a Linux server? Tough, pay up for more licensing fees.

Eventually apdip.net will be moved to IOSN servers. Either way over 7 years worth of ICT4D material will continue be available with the same urls.

2007-10-24

Auto reply the Unix way

Filed Under:

Don't you just hate those auto-reply emails to mailing lists and such? Auto-replies are still important though, especially if you are on extended vacation leave and want people who are trying to contact you to be aware of it.

The Unix way to do this is through the vacation(1) command and formail or in my case procmail.

vacation(1) is smart in the sense that it only provides replies once, and keeps track of this in a .vacation.db file in your home directory. Your auto reply message is a plain text file also in your home directory as .vacation.msg

From: kaeru@inigo-tech.com (Khairil Yusof)
Subject: At the beach suckers!
Precedence: bulk

I asked that there would be no salt in the margerita.. but but
there is salt in it. 

I could burn this resort down you know.. sir.. sir.. 

The bulk header will be respected by mailing lists and the auto reply will not be sent.

For your procmail recipe, just add (check your paths) and username.

:0 c
| /usr/bin/vacation username

The c will tell it to continue and go on to other filters after it so you will have a copy of the original email, or forward it etc.

When you're back, just remove the files and procmail entry.

2007-08-28

loading multiple files that match a string with vim

Filed Under:

loading multiple files that match a string with vim using command line unix tools

Using the ever useful grep, with -l option, it outputs only the matching filenames:

grep -rl --exclude '*.svn*' CMFCore *

vim loads multiples file in one line separated by spaces. So I need to join the multiple lines into one, separated by space. Here we have the paste command. For each line, put a space as the separator:

paste  -s -d " "

Finally I wanted vim to load this in GUI mode with tabs, so we put it together:

gvim -p $(grep -rl --exclude '*.svn*' CMFCore *  | paste  -s -d " " -)
  • the part inside the $( ) is the output of the command.
  • using pipes | the output of grep (the list of files) is read by paste
  • the - in paste is telling it to input the stdout output of grep

I then quickly have my editor setup for work,

http://kaeru.inigo-tech.com/blog/blog-images/gvim-tabs.png/image_preview

Powered by Plone CMS, the Open Source Content Management System

This site conforms to the following standards: