In this presentation, I will be focusing on how to use Solr as a kind of NoSql solution for Big Data. Topics will include scaling Solr both up and out, sharding, replication, caching, SOA, indexing, and synchronization. I will also give advise on how best to integrate Solr with other open source technologies such as Jetty, RabbitMQ, Spring, Ehcache, and HOWL.

If you are attending this conference, then I hope that you catch my presentation. If not, and you are interested in Solr as a NoSql solution, then be sure to check back with this blog topic as I will include links to the published slide deck afterwards.

Update May 14: Back from the conference. Many Apache commiters talked about Lucene 4. SolrCloud is Solr on ZooKeeper. Good keynote from Hortonworks. Lame keynote from Microsoft. O’Reilly published a nice review of the conference too.

Most of Tigase’s core code is written in Java. The best way to customize your version of Tigase is to build a Java project that extends Tigase classes or implements Tigase interfaces and to include the resulting JAR file in the libs folder. You tell Tigase about your classes through the init.properties file. There are over seventy different ways in which you can extend or enhance chat server functionality this way in Tigase but I am going to cover just the top ten here.

AuthRepository

Implement this interface to customize handling log in requests beyond what a single stored procedure call can do. If all you need to do is provide your own stored procedure, then stick with the TigaseCustomAuth implementation and specify your own stored procedure in the init.properties file.

AbstractMessageReceiver

Extend this abstract class to build a component. The most important methods to override are processPacket, getDefaults, and setProperties. The processPacket method is where you handle packets that are sent to this component. If you need to instrument your code with additional parameters, then use the getDefaults method to obtain your configuration out of the init.properties file. The setProperties method allows your component to accept dynamic properties that are set while Tigase is running.

XMPPProcessor

Extend this abstract class to build a plugin.. The Id method returns a string that must match what is in the –sm-plugins comma delimited list in order for this plugin to get loaded. The supElements and supNamespaces methods let Tigase know about the kinds of packets that this plugin supports. The init method is when your plugin gets initialized with configuration information from the init.properties file. The process method is what gets called when it is time for your plugin to process a packet.

What is the difference between a plugin and a component? Think of plugins as the best way to extend existing functionality or add new functionality to already existing or new stanzas no matter who sends or receives them. Components permit you to add server-side chat bots who respond to traditional stanzas such as IQ, presence, or message packets. Plugins register for particular stanza types and get invoked no matter who is the recipient. In order for the component to process a request, that request must be addressed to be sent to the JID of the component itself. When a plugin handles a request, it gets access to either the sender’s (if outbound) or the receiver’s (if inbound) session data. A component has no access to user session data when it is invoked. A plugin always runs in the JVM of the session manager so the only way to scale out plugins is clustering. Components can be configured as either internal or external which is a more natural and intuitive way to scaling out additional functionality. New in version 5.1.0 of Tigase is the ability to cluster external components.

DataRepository

In all likelihood, you won’t need to implement this interface. Objects of this type help you manage multiple, pooled JDBC connections and their associated prepared statements. The factory method for objects of this type is RepositoryFactory.getDataRepository.

ShutdownHook

Implement this interface if you need to run some code when Tigase shuts down but that code can’t take a long time to run. You have to add code to your project that registers your own shutdown hooks. Logging during shutdown time is already turned off so debugging this code will not be easy.

UserRepository

Implement this interface when you need to integrate Tigase user data (e.g. rosters, vcards) with your web site. If you still need to access a relational database, then consider extending JDBCRepository instead. If you are setting up a federated chat system, then you will also need to extend UserRepositoryMDImpl for multi-domain rosters, etc.

Here are some additional tips and tricks to customizing Tigase.

If you want to filter packets, then extend XMPPProcessor and return true from your override of the preProcess method. Returning true means to filter this packet out. Returning false means to permit this packet to continue.

If you want to count packets, then implement the poorly named PacketFilterIfc which is more suitable for statistics generation than for filtering. Be sure to override the getStatistics method and add your counters (name and value) to the StatisticsList argument. Your statistics will be included in the results to any ad hoc stats command request.

If your processor is more about extending existing functionality (such as integrating chat with your web site) than it is about providing completely new functionality, then consider extending one of the already written plugins from the tigase.xmpp.impl package. For example, extending tigase.xmpp.impl.Presence permits you to override the processInSubscribe, processOutSubscribe, and stopped methods for a more easier way to handle those complex changes in presence state.

You can also customize Tigase by adding custom scripts to the scripts/admin folder. Usually written in the groovy programming language, these scripts get loaded at Tigase start up time and are available to be invoked as XMPP add hoc commands. You can also reload these types of scripts dynamically without having to restart Tigase. Groovy code runs much slower than Java so you should limit the frequency of your usage of these scripts. You load these scripts either by copying then in to the scripts/admin folder and restarting Tigase or by loading them dynamically via the add-script ad hoc command. When copied into the scripts/admin folder, each script must contain AS:Description, AS:CommandId, and AS:Component in a comment block near the top. The component must match one of the item names returned from a disco#items query.

There you have it. Tigase is a highly customizable XMPP server that provides an excellent, scalable platform for real-time communications that is easy to extend to meet your specific needs.

Modern applications need to be configurable. What good is an application that cannot work after moving the database or any other dependent service? I find that applications tend to collect quite a large number of configurable information over time. When the project manager can’t make a decision, the engineer makes it a configuration switch to defer the decision till later.

This information doesn’t change during the run of the application. It just needs to get loaded into memory at start up time in order to be used during the run of the application. This is true for both client and server applications.

You don’t need a database to store this information between runs. That would be overkill. Besides, your configuration information is most probably manged by configuration management software which knows how to write text files. In all likelihood, your configuration should be stored in a text file.

Which format should this information get stored as and how should the information get read in and parsed? That depends on how the configuration information is organized. What are the informational complexity needs of the configuration? In my experience, that comes down to two choices, a bunch of name/value pairs or something that is context sensitive.

What does context sensitive mean in this case? It means that actual structure of the information depends on its earlier content. For example, I recently wrote a light weight configuration management system where configuration files get generated using a model driven template approach. The model can look something like this vastly simplified example.

class Server attribute port;
class MySql extends Server attribute password template mysqlconf;
object db instanceof MySql attribute port : '3306', password : 'test' ;

You could say that this could be captured as a series of name/value pairs but what the actual names are would depend on the proceeding class name (identified by what follows the instanceof keyword in this case).

The easiest way to parse something like this is to use a grammar parser generator such as Antlr. You  specify the grammar and tell antlr to generate the source code needed to parse input in the format of that grammar. Throw in a little boilerplate code to invoke the  parser and you get an easily traverable object hierarchy that contains the information to your specification.

This is easy to code and runs quick but it requires that you understand compiler theory enough to write that grammar specification. That’s not for everyone so I don’t recommend using Antlr if all you have is a set of name/value pairs. What should you use in that case?

It all depends on whether or not the name/value pairs are flat or in a hierarchy. Flat name/value pairs would  look something like  this.

dbport=3306
dbpassword=test

For Java developers, the best way to load this kind of information is with the Properties class. You call the load method to read the file and the getProperty method to access the value of any property. You can also enumerate through all properties and use the Properties class to update the information back to the file if need be. This is, by far, the simplest and easiest approach. A lot of applications can safely use this approach without compromise.

Hierarchical name/value pairs would look something like this.

Db:
       - port: 3306
         password: test

The three most popular formats for storing hierarchical name/value pairs is XML, YAML, and JSON. Historically speaking, XML would be the most popular choice for Java developers but there are also good libraries available for parsing JSON in Java too. YAML is more popular amongst the Ruby and python programmers  but there are parsing libraries for Java too.

With XML you get two flavors of parsers, SAX and DOM. SAX is an event driven approach where you code handlers that get called as the XML is being parsed. DOM does all the parsing up front then returns an object hierarchy for you to traverse in your code in order to access the underlying data. SAX is usually quicker to run than DOM but more complicated to code for as you will most probably have to maintain an explicit stack during the parsing operation. If you prefer the DOM approach but want to provide your own classes instead of using the generic xerces classes, then take a look at either the Apache Digester project or JAXB.

You have probably noticed by now that the kind of system level configuration for a typical application (client or sever) could be modeled as context sensitive, flat name/value pairs, or hierarchical name/value pairs. The question isn’t so much about mandatory organizational style as it is about what is most likely to make the most sense cognitively to the engineers who will be maintaining this application in the future? To me, if it looks like a catalog of system resources, then it will best fit the context sensitive approach. If it looks like default properties with some specific overrides, then it will best fit in a hierarchical name/value approach. If we are just talking about how to connect to a small list of services, then the flat name/value approach would be my preferred approach.

Depending on the complexity of the information in your app’s configuration needs, choose the right technology for the job. In this topic, we covered three such choices; Antlr for context sensitive configurations, Properties for flat name/value configurations (most popular), and XML/YAML/JSON parsing for hierarchical name/value configurations.

Maybe it is just the inner entrepreneur but I really enjoy getting to explore an innovative idea and bringing it to the light of day. That was perhaps the main reason why I enjoyed working on the Conversational Content Management beta so much.

What was innovative about this project was in the way that the collaboration problem was addressed. In most online collaborative document creation environments, the actual editing is the visual equivalent to what was used to be called half duplex. If you move the mouse and started typing, then you rudely took away control from whoever was editing last. This has a very chilling effect on the fragile process of group creative interaction.

Have you ever served on a board of directors? The monthly board meeting usually looks like this. A group of people discuss the agenda while an assistant takes notes. Later on, the highlights (i.e. the actionable summary) get published.

What if you could have access to that assistant in any meeting? That is what the Conversational Content Management beta is all about. All collaboration is done in the chat room itself. A robot records the entire conversation and extracts out the actionable summary later. The following technical white paper discusses how and introduces the following open source projects used to create this beta and accelerate its time to market.

• Hippo CMS is an open source Content Management System written in Java. Its emphasis is on publishing with a limited number of authors with access available to the anonymous public at large.
• Strophe is an open source Java Script library that permits web pages to access an XMPP chat server via BOSH.
• Code Igniter is a PHP MVC framework for writing web applications hosted by the Apache web server.
• Jack Rabbit is an implementation of the Java Content Repository which is how the chat reporter publishes content in the CMS.
• Tigase is the open source XMPP server, written in Java, that serves as the chat room implementation.
• Thrift is the communications layer between Tigase and the chat session manager.
• MySql is the open source relational data base where the web site and the chat server store their data.

Conversational Content Management

In the past two blogs, we have surveyed some popular logging and data selection techniques. In this blog, I will introduce you to some tools to graph or report the data in ways that make it easy to draw action oriented conclusions.

A picture is worth a thousand words as they say. Plotting a graph is a great way to spot trends or reveal new relationships in various aspects of your app. Graphing data entails inputing one or more CSV files and rendering an image based on the data from those files. I introduce four ways to do this; spreadsheets, OLAP, R, and Weka.

Microsoft Office, Oracle Open Office, IBM Lotus Symphony, and Google Docs all offer a spreadsheet application capable of graphing the data from CSV files. Typically, this entails opening up the CSV file which displays its contents in rows and columns. Then selecting the right rows and columns (do include label data for the graph) and invoking a chart wizard to render the image. After that, you capture the image, usually through some kind of print screen capability.

Sometimes your analytics is so complex that it is too cumbersome to use CSV files. Sometimes, the CSV files are just too big to import into the spreadsheet. Perhaps your analysts feel more comfortable slicing and dicing their data in real-time and pivoting with a drag-and-drop interface. For that reason, you may want to take a look at using an OLAP database as the back-end to your spreadsheets instead of opening up the CSV files directly in the spreadsheet app itself.

When analysing logs files the normal way, you have multiple data sets and they are all two dimensional. With OLAP, you have only one data set but it is multi-dimensional. This multi-dimensional data set is usually referred to as the cube. Most OLAP cubes have about 20 or so dimensions to them. Each dimension is actually a hierarchy of values which is how you can roll-up or drill-down in an OLAP spreadsheet. You still use the CSV files only now you load the cube with the CSV files then access the cube with spreadsheets. Most OLAP technology these days use a relational database under the covers.

Most of the relational database vendors also provide an OLAP offering. Microsoft and Oracle are the two most notable examples. Oracle’s offering is most probably the most mature as they acquired Essbase from Hyperian who, in turn, purchased it from a company of the same name. For an open source OLAP database, consider using Pentaho’s Mondrian project. The GUI of choice for both the Microsoft and Oracle offerings is MS-Excel. Mondrian is intended to be embedded in your Java applications and has a web interface too but no first class plug-in for MS-Excel.

Mondrian is open source and so lacks any licensing fee structure. Jaspersoft also releases a community edition of their OLAP server. The others have somewhat high licensing costs. Be prepared to incur high operational and maintenance costs when using any OLAP solution.

The down-side to using spreadsheets to render your graphs is that it is a manual operation. This can get to be a bit daunting when you have to generate these reports often. Another approach is to use the open source R Programming Language to generate the actual graphs.

Perhaps the biggest advantage to R is that you can easily apply various transformations to the data set, or work with multiple data sets, as easy as writing a formula. R comes with lots of plotting and graphing options such as dot charts, bar charts, pie charts, notched box plots, histograms, topographic maps, and heat maps.

Just like with the spreadsheets, you read in a CSV file’s contents to create a data set in R. Once loaded, there are various functions that you can apply to a data set in order to pivot, slice and dice, filter, normalize, etc. Vector and matrix math is available in R as are lists, data frames and just about any probability distribution function that you could imagine. The R programming language was originally designed for statistical analysis.

If you are trying to find correlation between various dimensions in your data, then R has a special feature for you where it automatically plots a 2D graph of every column vs every other column in your data set. This is called the scatter plot. You can quickly inspect each graph visually in order to discover hither to unknown relevance.

Ubuntu users can easily install the R programming language environment via the Synaptic Package Manager by marking the r-recommended package for installation.

Perhaps you see analysis in terms of data mining but don’t feel comfortable with using a programming language such as R. In which case, take a look at Weka which is a collection of machine learning algorithms for data mining tasks. Weka contains tools for data pre-processing, classification, regression, clustering, association rules, and visualization. There is a GUI tool that is easy to use. Although it prefers its own ARFF format, you can load Weka with CSV files. Like R, Weka can easily show you scatter plots to help you find correlation between various dimensions in your data. Unlike R, Weka provides advanced classification and clustering capability through the use of machine learning where you provide special training data that it uses to do a better job on the real data later on.

There you have it. With these three blogs, you now have a survey level introduction to the various tools and techniques for logging, selecting, and graphing your app’s activities for the purposes of analysing its performance and behavioural characteristics.

Only through the process of constant and unrelenting improvement can you hope to remain competitive. Analytics provides the reality check for that process.

This blog is part 2 of 3 parts. In today’s blog, I will go into how to massage the raw logging data captured from part 1 in preparation for graphing or reporting on interesting trends which you will learn about in part 3.

You have now instrumented your app for logging and have gotten all these raw log files with the collected activity of your app. These files capture all aspects of your app’s activity which is information overload when trying to spot trends or to answer specific questions about relevance or usability. What you need to do in the selection phase is convert those log files from a logging appropriate format to a graphing appropriate format.

converting log data into trend data

How do you filter and aggregate these raw log files into data that is more easily able to be graphed or displayed as a table based report?

If you are using Linux or Mac OS X, then you most probably have all the command line tools that you will need to get the job done. Windows users should check out cygwin which provides similar command line tools for Microsoft compatible machines. Here is a gentle introduction to some of these tools and what they can do for you.

If you are looking only at one type of activity and wish to filter out the rest, then consider using grep which is a general regular expression parser that can report on or extract out strings from your log files that matches patterns that you specify. It can also do some rudimentary aggregation and perform these operations on just one file or on multiple files in a folder hierarchy. Another way to aggregate is to pipe the grep output into the wc -l command.

If each line in your log file looks like a multi-column table where each column is delimited by a special character (e.g. CSV or Comma Separated Values files), then you can use the Linux command line tools cut, join, and paste to subset and union these rows both vertically and horizontally. The cut command extracts specific columns out of a file. The paste command combines two files together on a line-by-line basis. The join command combines two files vertically over a specific column.

Sometimes you might need to massage a log file in order to get it into CSV format. In which case, sed might just be the right tool for the job. This line oriented text editor is ideally suited for shell scripts that need to do this kind of data massaging.

For more complex aggregation, pivoting, filtering, and report generation, consider using awk which is a very advanced report writing tool. To use awk, you write this script that looks like a hybrid of the C programming language and a sed script. This script is fed your log file which it parses to generate the output.

Each of these tools perform a very specific task. You can easily combine these tools to get the entire job done through use of such shell programming features as pipes, sideways pipes, and redirection. Constructs such as conditionals, for loops, and here files should also come in handy.

A pipe takes the output from one tool and makes it the input to another. A sideways pipe takes the output from one tool and makes it the command line arguments to another. If that output is multi-line, then you will want to pipe it into xargs. Redirection permits a tool to get its input from a previously existing file or to save its output as a new file.

If your filtering needs to be context sensitive, where a match of one pattern should occur only in the context of another (possibly on a previous row in the log file), then consider using ANTLR to perform these more advanced filtering operations. It is easy to install ANTLR. Just download the JARs and run them. You will most probably also want to use antlrworks to create your grammar specification files.

ANTLR is actually a general purpose parser generator but you can re-purpose it via the use of its filter and rewrite grammar options and its integration with another Java library called string template.

Maybe you already use relational databases and would just like to take advantage of the expressive power of the Structured Query Language’s select statement to generate reports from your log files instead of all these different command line tools? If that is your preference, then consider dumping your log files into Hadoop and use Hive to run SQL statements against your logging data. While not standards compliant, Hive SQL is very expressive and will work on extremely large data sets (though not quickly).

The easiest way to install Hadoop and Hive is Cloudera which is available to Linux users as a Debian package. You will have to add their repository to your Synaptic Package Manager repository configuration before installing.

It’s amazing just how easy it is to slice and dice your raw log files in order to get the data just right for graphing and reporting. Today, you got a glimpse into some of the more common tools used for these purposes. In the next instalment, you will see how to use more open source tools in order to graph and/or display this data in professional looking reports.

This blog is a three parter. In today’s blog, I will cover the basics of logging. How you can add code to your application that will log its activity for later analysis. In the next part, I will go into how to select the raw log data in meaningful ways. In the final instalment, I shall elaborate on how to plot or graph relevant subsets of logging data for the purposes of insightful analysis.

Logging is a necessary prerequisite to any kind of data analytics. If your application doesn’t log its activity, then you cannot analyze it later. The simplest way to instrument your app for logging is to write out what it is currently doing to the console. Various tools are available to take the output to the console and to split it up into many files over time. Splitting up the output into multiple files will make it easier to copy to another server. It is not recommended that you manipulate and analyze the data on the same server as where your app is running. If your app can run as a service, then consider using runit with its svlogd tool.

If yours is a web application, then a lot of logging is most likely already happening for you. Both Apache and IIS record each HTTP operation, the end point URI, and the return code automatically. From these log files, you can learn a lot about how your app is used but there are limitations. You cannot trace the identity of the user as he or she flows through your site with this type of logging. The only type of demographic questions you can answer with this type of logging is browser type or OS. Nothing in the user’s profile is available. Form post data is also not recorded at this level of logging. That return code is from the web server and not from your application. Your insert statement may have failed but the server still returned a response so the return code will indicate success. One easy way to analyze Apache log data is with the open source project awstats.

The most popular way to log web application activity on the Internet is Google Analytics. This is a free service, provided by Google, where you drop a little java script on each page. Over time, you can access your data via the Google Analytics web site where you can track such metrics as types of visitors (browser, OS, etc), where they come from (what sites are linking to yours), most popular content (including number of visits, unique visitors, and average time spent on the page), and most popular landing and exit pages.

Google Analytics also lets you set up conversion goals that you can also track. These are essentially a sequence of pages that you hope to funnel your users through. This is usually important for e-commerce sites to evaluate the effectiveness of their shopping cart.

When instrumenting your app to log its activity, take a look at using a third party library instead of doing the writing directly in your app code. There really is no reason to duplicate all that logging logic in a million different places in your app. Instead, take advantage of the object oriented concept of encapsulation by wrapping your logging logic in a class or two.

Java developers get this for free in the java.util.logging package. The Log class and the Level enumeration are what you will be dealing with the most. Use environmental settings and a configuration file to customize how and where all that logging gets saved including the number of files to rotate the logs through. Logging levels permit you to write copious data for short term debugging purposes or less data for long term trend watching and forecasting purposes. Advanced use may entail subclassing the LogFormatter class if you need to do fancy stuff such as aggregating data.

PHP developers may wish to use scribe to handle the mechanics of logging. Originally developed by Facebook and now an open source project, scribe is great for centralizing logs from large server farms. There is a thrift based API which makes scribe available to other development languages and platforms.

From a data store perspective, logging usually entails writing huge amounts of immutable data to be aggregated and analyzed later on. This sounds like a perfect application for Hadoop. Even if you do write your logs to files, a batch process can come behind and import those files into HDFS. Hadoop is distributed so it is immensely scalable with no single point of failure.

From the humble print statement to the system console to massively parallel noSQL data stores, we have surveyed the most effective and popular methods for logging your application’s activity. This is an important first step on the road to analyzing your apps performance and usability. Stay tuned for the next installment which is all about how to select and prepare all these raw logging files for analysis.

Compiler theory and language parsing is a pretty deep subject in computer science. You start with Chomsky’s Theory of Formal Languages and end up with parsing strategies such as LALR(1) and LL(k). It’s not for everyone. ANTLR makes it easy to develop and maintain your own parser for a language whose specification keeps changing but only if you have a solid grasp of the fundamental concepts. Otherwise, it is going to look a lot like voodoo.

I have used ANTLR before but this is the first time that I have used version 3. There has been a lot of improvements since the last version. One of the biggest improvements has been the change of parsing strategy from LL(k) to LL(*). Perhaps a small digression to explain that is in order.

A very important part to parsing is the ability to recognize strings that conform to the language specification from those that do not. You write the language specification in a kind of EBNF format (file type is .g in ANTLR) and run an ANTLR tool that creates classes in a variety of languages (e.g. Java, C#, Python) whose job it is to parse these strings.

What the parser does is break the string up into tokens (this is called the lexical analysis phase) then process those tokens. At the same time, it is traversing the language specification which is a set of production rules. If it finds a rule for the next token, then the string is still compliant to the language specification. Otherwise, it is not.

What happens if it finds two or more rules that match the next token? This is called ambiguity which is a bad thing. The parser needs to decide what the next rule is or it can’t proceed. The way to make that decision is to look ahead to see if any tokens further down stream can resolve the ambiguity. Grammar specifications with limited look ahead are harder to read.

That is why the move to LL(*) from LL(k) is a big deal. With LL(k) you have to specify the maximum look ahead and the bigger the number, the slower the parsing. With LL(*) you can have unlimited look ahead and the parsing remains fast thanks to two other features new in version 3 called memoization and backtracking.

Some other cool features in version 3 is the ability to use ANTLR to easily create filters and rewriters. With these new features, it is easy to use ANTLR to create log file analysis tools, for example. Grammar rules can now have return values which allows you to easily propagate relevant information up the call stack of the rule traversal in a parsing session.

What do you do with a string once it has been parsed? Earlier versions of ANTLR allowed you to either embed your own commands into the parser class that get executed when the corresponding rule has been chosen and to create an AST or Abstract Symbol Tree that can be traversed after the parsing has concluded.

Another new feature in version 3 is a third option which is to use string template technology to generate output; however, that has not proven to be so useful in my own exploration of version 3.

Another thing that I did this past weekend was stress test the servlet using Tsung. I wanted to see how well ANTLR could perform when expected to serve about two requests per second for an hour.

What I found was that ANTLR was easily able to handle the load taking around one millisecond to perform the parsing. A surprise discovery for me what that the AST version performed just as well as the embedded commands version. I had expected that the AST version would take longer since you had to traverse the AST data structure afterwards. There was no noticeable difference in latency between the two types of parsings.

Scalable Server Side Language Parsing

There are many IDEs that are available including Visual Studio and NetBeans; however, the IDE that I want to talk about today is Eclipse. Originally IBM’s replacement for the atrocity known as Visual Age for Java, Eclipse has grown into a highly pluggable IDE for just about any software development language and platform.

It’s that high pluggability that makes Eclipse so special. That and the rich marketplace of freely available, high quality plugins that can be surfaced within Eclipse. Here are some of the plugins that I use on my home machine.

There are lots of plugins for Eclipse and it would be hard to set up just what you needed if you started with a blank Eclipse, just the shell with no plugins. That’s OK because there are lots of convenient packagings of plugins for you to start from. Some of the best packages feature eclipse for Java developers, J2EE developers, PHP developers, C++ developers, and modelers. Web developers might want to take a look at Aptana, especially if you do Ruby on Rails.

The good news is that you can mix and match. You can have multiple versions of Eclipse that you can enhance with additional plugins. I could have both the JDT and the PDT running as two separate instances of Eclipse or I could just install the PHPEclipse plugin into the JDT. Why pick one over the other? Use PDT if you are a full time PHP developer and pick PHPEclipse if you mostly do Java and just need to do a little PHP coding every now and then.

If you find that EMT is overkill for your design needs, then check out the Eclipse plugin for UMLet.

Another favorite plugin of mine is Pydev which is helpful for Python or Django developers.

Not only are there plugins for different development languages, there are also plugins for different frameworks. The Eclipse plugins for JDO, Spring, GWT, and GAE are some of my favorite in this category.

Eclipse plugins are also very helpful for more code related activities than just coding. Subclipse is a full featured plugin to interface with the Subversion control system. M2Eclipse allows Eclipse developers to easily build their projects and manage their project dependencies using Maven2. With Mylyn, you can access and keep track of your list of development tasks right within the Eclipse GUI. There are a lot of Mylyn connectors for various project management tools including Rally, CollabNet TeamForge, FogBugz, and Trac.

If you are a coder who prefers using an IDE, then consider Eclipse for its wealth of plugins that span a wide variety of languages, frameworks, and methodologies.

Surfacing Project Management within Eclipse

When setting up your own chat server, there are two open source projects that you should seriously consider; Openfire and Tigase.

Sponsored by such corporate leaders in IM as Jive, IBM, Apple, and Google, Openfire is an XMPP server written in Java and stewarded by Ignite Realtime.

Tigase has more humble and open source traditional origins where an engineer, seasoned in server development, had an “itch to scratch” and started building this XMPP server.

How are Tigase and Openfire similar? They both fully support the XMPP standard and many of the more popular extensions to that standard. They both parallel the high extensibility of the protocol with their own pluggable architecture. They are both written in Java.

You can extend the chat server functionality by writing your own plugins and components. The heart of the XMPP is in its three main stanzas; IQ, presence, and message. IQ is mostly about roster management. Presence is how those away and available notifications get delivered. Messages are what you text back and forth with your friends. If you want to add new functionality to those stanzas, or create your own, then you should write a plugin.If you want to write a chat bot (i.e. a robot that you can exchange messages with), then you should write a component.

A JID (Jabber IDentifier) is how you log in to the server and how you address other users. They look a lot like email addresses. A component has its own JID whereas a plugin does not.

How are Tigase and Openfire different? Openfire provides a web based administrative interface which you can extend with your own JSP. You can add your own plugins by packaging them up in a WAR file and adding them through the admin interface. Openfire depends on being hosted by a J2EE application container such as Tomcat. Tigase is a stand-alone program that is executed by itself. The only administrative interface for Tigase is a text based properties file that you can edit before launching the Tigase service. To get Tigase to load your plugins and components, you just have to ensure that your JAR files are in the classpath prior to starting Tigase.

With Tigase, you can also plug in your own repository instead of depending on their relational database. Tigase is more oriented towards high scalability with its support for clustering, virtual hosts, and thread management.

Which one is best for your needs? If you are more concerned about IT footprint and cost containment than you are about serving large numbers of people, then you should start with Openfire. If you already know that you are going to have to serve large numbers of users and are willing to commit the resources of a server farm, then you won’t go wrong with Tigase.