Plone
2008-11-17
One Night in Bangkok
Back in Bangkok and working at the UN building for another week, but this time at (ILO) instead of UNDP and doing technical development work instead of a programme/project role before. In the short term, it's quite enjoyable and it gives me opportunity to work on my technical skills again. Doing technical work full time also makes me realize that there is also a wealth of UN type programme work which I'm also good at and also enjoy doing. Letting almost 4 years of UNDP regional programme experience go to waste is also not good. So I'll need to figure out how to get jobs that balance the two skills.
The technical work I do, involves using Plone to quickly create custom content types, some UI changes and also simple online forms for intranet, for thing such as reports. Where Plone helps in this regard is that it already has a framework for contents and for user created views. This is where users (or content managers) can quickly create their own custom reports through a web interface. eg. I want all my reports, involving all my missions in specific countries of Thailand and Cambodia, from January to June, related specifically to Child Labour. With Plone, if the content field types are indexed and registered properly, a none-technical user can with several clicks quickly create a dynamic search view for themselves. I'll be sharing a detailed how-to on this based on my development experiences. More on this in a separate entry.
Place I'm staying is great, it's called Centre Point, and for quite low rates, you get serviced studio apartments, complete with kitchen, fridge etc. If you have to stay for a few weeks, it's much better than a hotel room. Most important of all? Unlimited free broadband access to rooms.
Another great thing about Bangkok, is the the density. This means that there are lots of little shops packed in every street providing various services. Laundry, convenience stores, massage (great when you're tired from travel/work), food etc. You can easily get access to them within a few minutes walking distance.
3 weeks (+3 days in Penang soon after I get back), is a bit of a long stretch for travel though even with all the conveniences. In future, I'm going to need to keep to a max of 2 weeks at most.
2008-09-19
FOSS Knowledge Transfers
FOSS developers have a huge advantage over proprietary software developers in that you have access to the development of the tools and libraries you use. This includes bugs, project direction and insight from the main developers.
I'm getting back into development working with Plone and Zope3. So here's how I go about it, and like most others, there isn't anybody around me working with it.
I don't want to simply know how to use the tools, and libraries to write basic applications. You should be able to do that with the docs and tutorials provided by most projects. I want to understand best practices and the design of them. At first, to at least understand what the direction of the next version will be, and then to be able to contribute towards development and direction of the next version. Even if it's just at bug reporting and feedback on proposals.
Why is this important? Most serious projects need to be maintained and improved. If you're using frameworks and libraries, then if your project is not in line with upstream, you'll be losing more and more of the benefits of collaborative development of FOSS. You may end up duplicating work, or not be able to take advantage of the new features in new versions. If you don't even file bugs, then if nobody faces the problems, maintenance of them is all on you.
You also want to tune in, because there is a lot of tips, links and other experiences that other more experience developers share. You can then learn more about what tools they use, and their development environments.
So how do I start?
Subscribing to Mailing Lists
I will usually subscribe to a few project related lists:
- User lists if it's also used by end users, and for end user questions
- Developer lists (general and also components I'm interested in)
- Commits lists - a list of latest changes in the project code repo
The user lists is where you may spend a lot of time in, initially for basic knowledge of using the tools and libraries. It's a quick way to gain answers and insights to common questions, simply by lurking. You're also likely to get a favourable and insightful response to questions about basic understanding on how things work.
Developer lists provide a wealth of information on various bugs, design decisions and build failures of unit tests or compilation. This is where you get a better understanding on what's wrong or lacking with the tools and libraries you're using, and what the develoeprs are doing about it. Knowing about the problems in advance is also helpful for your own planning. Problem A is not going to be fixed until next major version, and since it effects my project, I'm going to need to do a workaround for now.
Commmits lists, are all the changes that are done to main repository and may also include latest issues. For large projects, there may be quite a bit, but usually a quick view of it, will allow you to grasp if there is anything of interest. I also find it fast to search locally and have ability to check revisions offline.
Wow this is a lot of email. Well you can filter them. I do it server side via procmail, others use gmail labels. Whatever works for you. I'll get back to how to actually read them later.
So for zope3 I have
- zope3-users
- zope3-devel
- zope3-checkins
Developer project pages
By this, I usually mean something like Trac, Launchpad or Sourceforge. Worth to bookmark this on your toolbar, as you'll need to access it often. These pages will give you a summary of the project, including milestones, design/feature proposals and most importantly a place track issues and make comments.
Planets
The collection of developer blogs is an awesome resource for understanding what more experienced developers are thinkinga about. From how they deal with some code problems or bugs, build development, tools they use, events happening, or just plain none-technical stuff.
For example in today's http://planetzope.org/ there is a quick comprehensive overview of Zope's build environment. "I want a pony: Django Cheeseshop".
An an example of none-developer related but interesting entry:
Kill Your TV by Alexander Limi
IRC
Of course there is IRC channels. These quite often are idle, but sometimes worth asking for quick directions or pointers when Google doesn't help. More importantly, you can participate in developer workshops/sprints virtually even if you can't be there, by being in the channel at the time.
Other Sources - Screencasts and Conference recordings
If you have access to broadband, these are also great resources. Most of us here in Asia are unlikely to be able to afford to attend conferences in Europe or North America where most of the developer conferences are. Luckily a lot of the presentations are recorded and we can learn from them.
Allocating Time
You (and project managers), need to be able to allocate time to be able to get the benefits of the above resources. A bad way to do things is when there are no branches. Everything is in trunk. Large projects run this way, are day to day things, badly planned and living on the edge. Quite often, they end up being abandoned as the effort to sync it to latest features in new framework (or worse.. even security fixes) become too difficult as you're 6 months behind.
Most FOSS projects are on a 6 months cycle. So it's worth developing on two branches, stable and next version. When next version of your framework/libraries are released, you'll be ready. It may also even include fixes to the bugs you had with previous version since you've submitted them either as an issue or an issue with patch. You're also able to learn and be involved in the development version. Repeat cycle. Time then needs to be allocated, to allow developers to be able to work on both branches, with priority of course on current stable branch which is usually in production use.
As for all the blogs, mail and so on. Some of them you'll deal with in day to day work, but some information such as links to video podcasts or a large patch may take longer than time available in your daily schedule. For projects I manage, I usually make sure that there is a few hours of "idle" time for people to allocat some time to reading, learning or to hack on some interesting problems. Set aside this time once a week at least to be able to keep your skills sharp and up to date. Also don't worry too much about not being able to read everything in the mailing lists, just make sure you've set aside this time each week to stay up to date on constant basis.
2008-01-03
Tesseract
Cleaning email, I ran across Fred Norohna's post regarding Tesseract.
It's an open source OCR command line tool, that takes TIFF files and converts it to plain text. So I decided to have a go at it with the cover of the Twisted book.
Original tif image here:
http://kaeru.inigo-tech.com/blog/blog-images/twisted.tif/
The resulting plain text looks like this:
Twisted Network Programming Essentials Twisted is an o en source network—a lication framework written in the P thon programming language. As elegant as Python, Twisted brings power and flexibility to Python programmers. Because Twisted is asynchronous, your programs won’t lock up while waitin for a res onse from the network. It’s also event—driven—'ust write code 8 P 1 for the network events you want to handle and let Twisted do the rest. A dedicated corps of developers work hard to keep Twisted extremely stable and secure. In Twisted Network Programming Essentials, author and experienced Twisted developer Abe Fettig starts you on the journey to Twisted programming mastery. Begin by downloading and installing the Twisted framework. Next, progress to sending TCP/ IP messages between simple clients and servers. And then the fun really begins! Twisted can do far more than send and receive messages using low—level protocols such as TCP/ IP. The Twisted networking framework makes working with higher—level protocols a breeze. This book teaches you how to; - • Use the twistea.web.client module to access web pages and web services via HTTP • Use Twisted to write your own web server • Email—enable your applications, using Twisted to send and receive email via POP5, IMAP, and SMTP • Develop web services using Twisted’s support for XML—RPC and SOAP • Use the twistealconc/0 package to write custom SSH servers and clients • Secure your network applications using the twisteacrea authentication package One of Twisted’s great strengths is its comprehensive support for a wide range of protocols. \X/ith Twisted, you can focus on what you want to do and on the application you want to deliver, with- out having to implement the underlying network protocols. The Twisted framework developers have done the heavy lifting for you—all you need to do is stand on their shoulders. Abe Fettig is a software developer specializing in network communication and integration. He writes software that helps messages and data flow between people and computers. Abe has been developing with Twisted since 2002 and has used it to run production web and mail servers. Glyph Lefkowitz is the creator of Twisted and wrote the foreword to this book.
That's quite accurate. What this means is that scanned documents usually already in TIF format, could with a little bit of code also be full text indexed in Plone. Let's see if I can get this working this weekend. In addition to plain text, legacy word docs, opendocument text and pdf, this would satisfy quite a lot of basic document search requirements.
Pity however that Firefox doesn't have TIF support built in, then we would be able to have image previews for the browser too.
Don't Hate
Here's a nice article from Martin Aspelli, who wrote Professional Plone Development book and contributes a lot to Plone. He makes a lot of good points that could apply to other FOSS community based projects.
It provides nice insights on different views towards community projects, and how it can affect the individuals working on them.
So for those slamming the organisation of say the community LPI exams due to some administrative issues with LPI or something, spare a thought for the individuals working to organise affordable community exams in their spare time. Community things like these are not for profit companies, to help improve things you have to be a part of it or support the community in some way.
Quite often you're benefitting for nothing but your time.
How much do commercial companies charge for MySQL training events?
How much does it cost to attend a MyOSS meetup where a MySQL employee is presenting?
Criticisms are good, but it's best with suggestions that shows good will.
Example,
"It sux, there is no food/drinks and any freebies"
Better,
"Since it's during dinner time, and we don't get much time after work to eat, how about we charge RM5/person so that we can have pizzas?"
Here are some upcoming activities:
- Jan 10th - Meetup MySQL/CouchDB
- Jan 26th - Accessibility Workshop (Visually Disabled)
- March LPI Community Exams - http://lpi.foss.org.my
2007-12-28
Efficient use of Resources
This afternoon, I finished replacing Apache httpd with Cherokee on Inigo's main server. Now Squid is handling access to all http services first, with Pound handling the management of the backend servers.
There really wasn't much need for Apache httpd, as we don't use most of it's features and without tuning, it takes up a lot of memory just to do rewrites, http proxy for the backends and logging.
Squid uses much less memory, and is very fast. Kagesenshi is working on tighter integration with Cachefu for the Plone sites. Even without that, you will find that sites like http://foss.org.my are now snappy (1.02 seconds total according to firebug). It's improved the speed of http://mirror.inigo-tech.com also which is a slow external USB drive.
The performance bottleneck now is actually memory. While FreeBSD's virtual memory does an awesome job (we're using 1201MB of swap at time of writing), we are now running 4 separate virtual servers, each running it's own self contained services. Not much more we can optimize now. Long idle processes of course, take several seconds to swap back in. So adding another 1GB of memory (total 2GB) will give quite a bit of breathing space and get rid of that lag.
For those that are curious, all this including http://www.apdip.net which used to sit it's own server is running on RM2.5K worth of hardware.
