Author Archives:
Eating Frogs: A Short Case for Incremental, Risk-Driven Software Development
“Eat a live frog first thing in the morning, and nothing worse will happen to you for the rest of the day.”
-Mark Twain
If you have ever been a software project manager, business analyst, developer, or purchaser of software development services, how many times have you encountered a project with the following characteristics:
-
All the requirements are done up front, with an emphasis on getting them “perfect.”
-
Design follows the same path as requirements – all designs and mockups must be complete before development proceeds.
-
In the project plan, coding for the system is broken out into “iterations” – one piece of the code is built at a time.
-
Project managers and technical leads schedule developers to implement the easy, simple requirements first in order to show rapid progress and build momentum.
-
When clients are shown the product at the end of iterations, they ask for changes that ultimately send shockwaves through the rest of the requirements.
At this point, most large projects are destined either to go significantly over schedule or over budget, or they fail outright – the project is scrapped. The reason for this is two-fold.
-
The Waterfall lifecycle is masquerading as an iterative and incremental lifecycle.
-
Management is pushing key risks to the end of the project.
The Rational Unified Process (RUP) is a generally-accepted solution to this problem. It provides a framework within which iterative and incremental software development can occur in a way that reduces risk over the lifecycle of a project. RUP is based on several principles:
-
Use-case Driven Development: Use cases guide the development of the system from requirements all the way through test. They also provide system use context far superior to list-based requirements.
-
Risk Driven Development: The most complex, significant, and risk-laden parts of the system are developed first. Over the lifetime of the project, risk decreases since the “harder” parts are done first.
-
Architecture Driven Development: Related to risk driven development, the early iterations of development focus on the parts of the system most critical to its structure, providing a framework around which the rest of the system can be developed with relative ease
-
Iterative and Incremental: The system is developed one small piece at a time with frequent end-user input. This ensures that the right system is being developed, and that there are no surprises at UAT time
Though RUP is a proven process, it can cause some indigestion for those unfamiliar with it – and especially those most familiar with the waterfall model. Some points of pain for RUP are:
-
Requirements are not complete before development begins
-
Initial progress can seem slow due to the addressing of high risk items early in the project; it takes longer to establish momentum
-
The process wreaks havoc on traditional project schedules – long range estimates for the completion of the project are best given towards the end of the Elaboration phase… after some development is already completed
On the whole, however, RUP presents far fewer risks that traditional waterfall development. Waterfall projects often veer wildly off schedule due to surprises near the end of the build phase – this is far more risky than RUPs practice of postponing long-range scheduling to the end of Elaboration. Waterfall projects may show lots of early progress as developers pick of low hanging fruit – but the benefits are fleeting; sooner or later the big risks are going to be encountered and, if they’re show-stoppers – all that low-hanging fruit was picked for nothing. Finally, requirements rarely, if ever, remain stable throughout waterfall development. While RUP may seem disquieting in addressing requirements in chunks throughout the process, this is really no different than what happens in waterfall – with the exception that RUP plans for the changes and is structurally built to handle them.
If you’re interested in trying this process, highly suggested reading is The Unified Software Development Process by Jacobson, Booch, and Rumbaugh. It’s dry reading, but full of wonderful and detailed information about how the process works.
Building New Bridges into NRC and USDA
Counterpointe is pleased to announce two new contract awards with the U.S. Nuclear Regulatory Commission (NRC) and the U.S. Department of Agriculture.
NRC and IWMS
NRC intends to implement an intergrated workplace management system (IWMS) with the ARCHIBUS facilities management system. Counterpointe will provide services that include:
- Analyzing an existing and heavily customized ARCHIBUS instance already in place, and determining a procedure for upgrade
- Refitting business processes to use standardized, out-of-box functionality present in ARCHIBUS with respect to space and asset management
- Implementing a new, personalized ARCHIBUS solution that minimizes heavy customizations and promotes an easily maintainable system
Once in place, the system will greatly streamline facilities management, asset management, moves, and much more.
USDA and UPC
The USDA has recruited Counterpointe for an operational support and adaptive maintenance contract with Food and Nutrition Service’s WIC program. Counterpointe will keep an exisiting UPC (Universal Product Code) system up and running as a new team of contractors redesigns and upgrades the organization’s UPC system to a new electronic benefits transfer, which is a key part of the USDA strategic goal to improve service to low income women, infants, and children.
Customizing ARCHIBUS Intelligently
One of the key strengths of ARCHIBUS as an integrated facilities management tool is its flexibility. The product offers a nearly limitless level of personalizations and customizations, allowing users to perform the simple (e.g. adding a new piece of data to a building) to the complex (adding entire new components and subsystems).
ARCHBUS’ flexibility is a tool, and like any tool it must be used carefully. Changes to an ARCHIBUS implementation generally fall into two categories:
Personalization – This involves minor changes to the system that do not require modifications or additions to system code. Examples of personalizations are:
- Adding, changing, and removing fields in forms
- Implementing custom roles and role based access
- Restricting report results to a certain subset of data
- Creating dashboards
Customization – This involves more substantial changes to the system, generally involving system code and the implementation of custom workflow rules, processes, and complex tasks. Examples include:
- Adding and removing steps in pre-existing ARCHIBUS workflows
- Integrating legacy products into the system
- Performing complex external system integrations
Personalizations are clearly the eaiser of the two to perform, and pose the fewest problems with respect to the system’s maintainability – particularly with respect to upgrades. Customizations, however, can easily pose significant problems if not performed carefully. Since these enhancements can involve changes to system binary code, core view files, internal system configurations and other key items, it is important to design and implement them intelligently.
We offer the following three bits of advice when it comes to ARCHIBUS customizations:
1. Increase your requirements time in your project plan by 25% – 50%.
Requirements will take longer to develop because you will want to review your business processes. In many cases, clients find that changing their own processes to match ARCHIBUS workflows (instead of the other way around) will save time and money not only in the software development process, but in the business processes themselves. ARCHIBUS processes and workflows are built upon industry best practices, and it could do your organization good to take advantage of that collective knowledge base.
2. Increase your design time in your project paln by 25% – 50%
If, after a requirements iteration you determine that customizations will indeed be necessary, allocate a very healthy amount of time to design. A rule of thumb is for project managers and developers is to estimate the time necessary to complete the design, then increase that number by at least 25%. The reason for the increase is that developers must not only perform the standard system design required for any enterprise software system, but they must also design for the remaining bullet points in this article.
3. Keep ARCHIBUS components and custom components seperate
Here lies the onion of why ARCHIBUS customizations can be so risky: Short project timelines lead to insufficient design time, which means that developers must implement customizations as quickly as possible. The quickest (but by no means the best) way to implement many customizations is to modify ARCHIBUS system code and weave customizations directly into the internal fabric of the system. In the short term, this is great: the project doesn’t take very long and the customer’s needs are met… until a year or two later when the client wants to upgrade the software. Now comes the pain.
When a new version of ARCHIBUS is released, its source code obviously isn’t going to contain your specific customizations. It’s going to be up to your developers to literally go line by line through all the customizations they made, and integrate them line-by-line into the new version’s code. Essentially, you’re doing all your customizations over again!
With sufficient design time, however, your developers can organize customizations into self-contained components that can be developed, deployed, and tested seperately from ARCHIBUS. Under this scenario, the upgrade process is much less labor intensive and error prone. Even better, the extra requirements and design time spent up front will actually pay for themselves during your first upgrade, because the upgrade takes so much less time.
So there you have it. Avoid significant customizations when possible – but when you must have them, allow plenty of time for your team to design a solution that will decouple core ARCHIBUS functionality and customizations. If you’d like more specifics, feel free to contact ARCHIBUS senior developer Chris Newman at cnewman@cpointe-inc.com.
Saying Goodbye to Sharepoint Folders
Allowing the use of folders in document libraries is one of the more unfortunate decisions Microsoft made with Sharepoint. Our love of folders is born of familiarity; they’re something we’re used to, comfortable with, and unwilling to let go of even if there’s a better alternative.
But for anyone that’s ever tried to backup a document library or email a URL knows just how much of a problem folders can be. Folder structures find themselves inserted into the URL of every document in your library – giving you atrociously long paths that are often too long to be saved on a harddrive.
In additon to the above, use of folders simply fails to leverage the full power of Sharepoint. Far too many people leave behind cool features like views and metadata in favor of the ‘tried and true’ folder system – only to become a.) frustrated by the types of problems mentioned in the last paragraph, or b.) asking themselves “why am I doing this online when I can get the same folder structure in a network folder, with much greater speed?”
Read my lips: “No more folders.”
Remember when Gmail first appeared? People initially found it’s ‘conversational’ model of organizing email to be frustrating and unfamiliar – that is until they realized how great it is. The same can be said of the proper way of organizing Sharepoint content, which is NOT to organize documents in vast, deep nests of folders.
Instead of folders, you should begin to think about two things: Content Types and Views.
Content types allow you to define bits of information that can be associated with documents. Combined with views, which allow you to view cross-sections of your document libraries based on information in the content types, you will be able to access your information more quickly and effectively.
For example, imagine you have the following simple folder structure in a document library:
Penskey Project
Meeting Minutes
Contracts
SLAs
Manhattan Project
Meeting Minutes
Contracts
SLAs
Instead of using folders, you could create a content type with two columns: Project and Artifact Type. You can then specify the project (Penskey or Manhattan) and the Artifact Type (Meeting Minutes, Contracts, SLA) at the time you upload the document. Finally, you can create any number of views in that document library so you’ll be just one click away from the documents you need. Create a view to show all Manhattan documents, another to show all Penskey Meeting Minutes, and another to show SLAs from both projects. You can imagine how this could be very useful when you have deeper folder structures.
Of course, creating content types takes a little more time up front and forces you to think carefully about the way you organize information… but that isn’t really a bad thing.
When it comes to Sharepoint folders, just remember: take a deep breath and let it go. Content types and views are the way to go in Sharepoint.
About the author: Chris Newman is a senior software developer at Counterpointe Solutions, Inc.
The Joy of Continuous Integration – Episode 1
This is the first of a multi-part series on continuous integration, featuring Hudson CI, by Counterpointe Solutions’ Michael Kim.
I recall a quick conversation I had with a friend and former colleague, Paul Duvall, who wrote a book about continuous integration (CI) a couple of years ago. I was not well versed in CI at the time, but these days I’ve been testing out a number of tools to incorporate the concept into our projects at Counterpointe. The open source CI tools available these days enable relatively easy and cost effective implementations. I have been researching two common tools – Hudson CI and Cruise Control – to compare the two to see which would work better in different situations. This is an account of the lessons I’ve learned.
No man is an island
Both Hudson CI and Cruise Control don’t do much on their own. To use the tools for greatest effect, you’ll need to have to understanding and experience with version control software like CVS or Subversion, and management/administration of applications servers(e.g. JBoss or Glassfish) and integration testing frameworks (e.g. JUnit, TestNG, Selenium). It’s a daunting list of things to know, but having that foundational knowledge will help you understand all the configurable items that Hudson provides.
When is CI appropriate?
If you’re working on a one-person project, a CI process is likely overkill. You should only attempt putting one together if you’re interested in learning the concept or putting together a pilot for a larger project. However when you go to projects with two or more developers working on the same set of code, or you have teams working on components with dependencies on one another, CI begins to show its benefits. When someone checks in compile-defective or run-defective code, it can have a negative cascading effect on productivity. The project may not be able to move forward until the author of the buggy component corrects the issue.
The three big issues CI can address
The first issue CI can handle is compilation issues. These can be resolved quickly by a CI process, which will checkout the latest code from version control, run a build, and notify the team of any issues.
The second issue is that of runtime issues at the unit-level. After compiling code from version control, CI processes can execute unit tests and report problems automatically.
Last, but certainly not least, CI can address integration testing by loading data via SQL scripts and executing test scripts against a database and application server that the application is running on – a vital tool for conducting regression tests.
Putting the pieces together
Beyond setting up Hudson CI itself, you will need to have version control set up so that Hudson can draw code from it. You will also require supplimental open source tools to integrate Hudson with your version control, such as Ant and build scripts to automate the compilation, build, and deployment processes. Finally, you will need both data-loading SQL scripts and a tool to control your application server. More details on these issues will be provided in later installments of this series.
That’s all for now. My next entry will step through setting up the environment to integrate Hudson with version control.
About the author
Michael Kim is a Principal Developer at Counterpointe Solutions, whose primary focus is on Java EE development process and system architecture.
Is Sharepoint Matching Your Needs?
About a week ago, I came across an organization suffering from the ‘collaboration blues.’ It was looking for a way to automate document processing, search for and contribute to documents across multiple team members, create team workspaces, associate metadata with its documents, and expose all this functionality over Intranet and Extranet. This set of requirements practically begs for a solution in Sharepoint, which is what I was going to recommend to the organization. Imagine my surprise, then, when I discovered this organization already had Sharepoint.
This situation is not at all uncommon. MOSS 2007 and its successor Sharepoint 2010 contain a set of out-of-the-box capabilities so large that it can become a liability. This organization’s Sharepoint instance is not the first I’ve seen relegated to the role of a glorified file server – many groups adopt Sharepoint without realizing the true power of content types, site templates, workflows, and other features that make a mature Sharepoint implementation a truly beautiful thing.
If you have a Sharepoint instance that’s looking a little too much like a file server, then take a peek at some of the items discussed below. They could be the keys to transforming your instance into something truly useful.
Content Types
Sharepoint content types allow you to associate data with different types of documents. Your HR department may create a content type for resumes that includes additional information like skill sets, candidate location, rating, etc. It’s very convenient to have this type of data in addition to the resume itself, because once you’ve created the content type you can search for resumes with a rating of ‘Excellent’ or ‘Project Management’ included in their skill sets. Furthermore, you can restrict the range of data that can be entered into these fields by using custom lists (this will keep one recruiter from listing a candidate’s rating as “Great” while another prefers “Excellent” – governance, discussed later, plays a big role here.)
Site Templates
Most people know that you can create any number of site collections in a Sharepoint implementation, but surprisingly few realize that you can save a “model” site collection as a template. If you’re making subsites for a number of project teams that will have similar structure, lists, navigation, etc., then there’s no need to create them from scratch each time. You can simply create the first site collection to your liking, save it as a named template, then use it as the site template for the next site you create. All the navigation and web parts will be in place, providing your collections with a consistent look and feel.
Workflows
Sharepoint is capable of routing documents through approval workflows that can be customized as you see fit, and it can generally be done without writing a single line of code. If you require processing for your documents, make sure you check out the basic out-of-the-box Sharepoint workflows before you go looking for a custom solution. You may be surprised by the flexibility offered by the standard workflows.
Other Cool Things
- You can create timer jobs to execute according to a schedule. This is useful for a number of things, especially synchronizing Sharepoint document libraries with external systems
- You can configure document libraries to receive emails. You can associate a document library with an email address and automatically populate that library with emails sent to its pre-configured address
- You can create custom web-based forms using InfoPath. InfoPath allows you to create forms without using code that can be deployed to Sharepoint and made available to users through the browser. Form information can be saved into document libraries or manipulated in practically any way you see fit, all without writing code
Governance
Governance is, in short, the creation and maintenance of principles that guide the use and implementation of Sharepoint to that it becomes a reliable, consistent, and useful system for the people it’s supposed to benefit. Governance is to Sharepoint what laws are to a nation. Sadly, this most important part of the business of running Sharepoint is also the most overlooked. The result is Sharepoint implementations suffering from one or more of the following:
- A patchwork of loosely related and often redundant site collections
- Inconsistent look, feel, and navigation throughout the site collections
- Non-existent or poorly defined content types that make locating documents nearly impossible, resulting in deep folder structures within document libraries that effectively turn Sharepoint into a file server
- No guidance as to how to implement and deploy custom features and other upgrades
- Lack of SLAs that cause Sharepoint maintenance windows to clash with project deadlines, among other serious problems
- A Sharepoint implementation that people hate to use
If any of the above looks familiar to you, then you probably have a governance problem.
I could go on all day about the robust features of Sharepoint, but we’ll leave it at this for now. If you have any Sharepoint issues you’d like to discuss, feel free to email cnewman@cpointe-inc.com.
Java EE Unit Testing Made Easier
The concept of unit testing has been around for some time and is considered a good practice for all developers. Frameworks for unit testing abound in the Java EE landscape, including such options as JUnit and TestNG. These and other frameworks are best utilized with POJOs, in which functionality to be tested can be isolated to a single method. This even works for complex, heavy-duty “algorithmic” methods because, ultimately, these methods can are isolated to a single class (if the principle of encapsulation is closely followed) and are independent of external resources such as EJBs, JMS, or database calls.
However, leveraging the full power of Java EE generally necessitates use of container-managed or other external objects with complex dependencies (e.g. EJBContext or JPA), making unit testing a difficult practice to follow. “Microcontainer” options are available and include products such as GlassFish and JBoss, but setting these up can be prohibitively cumbersome and platform dependent – GlassFish and JBoss microcontainers require Oracle and JBoss target environments, respectively.
After numerous frustrating attempts at performing EJB unit tests with TentNG and JUnit, there appear to be new developments on the horizon that will make this task much more simple. Arquillian is a new testing framework (currently in Alpha) designed to make the testing of EJB/JPA objects as straightforward and flexible as testing options for POJOs. Here’s to hoping this new framework makes life a little easier for us all.
Upserts in SSIS Using SQL Server Business Intelligence Studio
SQL Server Business Intelligence Studio provides an excellent graphical interface for creating some very complex ETL packages, but there are a few simple operations that are surprisingly absent. One of these operations is the simple upsert operation, where you want to either insert or update a record in the target from the source depending on whether or not it already exists in the target.
If you’ve already tried looking for an upsert transformation in the tool bar, you can forget about it – it simply isn’t there. There are two common ways to set up and upsert, one being the clean-but-slow way (using Lookup transformations), and the other being the fast-but-dirty way (using insert failures).
Method 1 – Clean and Slow
- Perform the requisite boilerplate steps by adding a new Data Flow Task, then add and configure your source to pull whatever data you need, including whatever key (or surrogate key) you will need to check if the record exists in the target.
- Add a Lookup transformation and connect the source’s output to it. Double-click the Lookup transformation, and ensure you set the ‘Specify how to handle rows with no matching entries’ dropdown to ‘Redirect rows to no match output’
- In the Lookup’s “Connection” menu, specify the connection for the data target (since you will be looking up the source data’s key in the target).
- In the Lookup’s “Columns” menu, link up the source key column with the target key column. Note that if the data types of the two columns don’t match, you’ll need to set up a Data Conversion Transformation between your source and the lookup. This is shown in the screenshot below.
- Drag a destination object (e.g. OLE DB Destination) onto the workspace and connect your lookup’s Match Output to it. You’ll point the object’s connection to the target and peform column mappings as needed. This path handles the ‘insert’ part of the upsert.
- Drag an OLE DB Command object to the workspace and connect your lookup’s No Match Output to it. You’ll set up an update command here, setting the columns in your source to map to the parameters specified in the SQL statement. This path handles the ‘update’ part of the upsert.
- That’s it! Below is a screenshot of the final product
This method is nice and clean, but the lookup can drag performance on large datasets. If performance is at a premium and you are a.) confident in the integrity of your database schema (i.e. you have good primary keys set up on your target table), and b.) are fairly certain that no errors exist in your data – like those pesky truncation or foreign key violations - then you may opt for a more simple version.
Method 2 – Quick and Dirty
- Perform the requisite boilerplate steps by adding a new Data Flow Task, then add and configure your source to pull whatever data you need, including whatever key (or surrogate key) you will need to check if the record exists in the target.
- Set up a data conversion task if you need one.
- Add your destination object, configuring it to redirect rows on error. Map your columns as needed, making sure that you map the keys of the two tables together.
- Add an OLE DB Command transformation, and connect the error output of the destination object to it. Configure your command object to update the target table where the key field equals the value of the key from the source.
- That’s it! Below is a screenshot of the final product:
The performance on this method is much faster, but this operation is a whole heck of a lot more dangerous. Any and all errors coming out of the destination object will be “swallowed” and you’ll never know what they are. Use this method only if you really know what you’re doing.
There are, of course, other ways to perform upserts (e.g. using Slowly Changing Dimensions), but I’ve found these methods fairly cumbersome. The two methods above get the run-of-the-mill upsert job done for me every time.
More questions about upserts or SSIS/BIS/ETL in general? Email the author at cnewman@cpointe-inc.com.
Evaluating Sharepoint 2010
In an effort to streamline internal business processes, among other goals, our team is undertaking a plan to internally adopt an on-site deployment of Sharepoint 2010.
We currently use Sharepoint 2007 hosted in the cloud by a third-party provider. This solution has worked well for a limited set of business needs, but a desire to extend it with custom features, integrate it with other enterprise applications, and have greater control over disaster recovery have prompted our team to evaluate bringing a solution on-site.
Our pilot will include a full-featured, two-server Sharepoint farm installed in such a way that it can be easily scaled to an n-server architecture that can take advantage of load-balancing and server clustering. The instance will be integrated with several internal applications – mostly HR-related – and then augmented with a number of out-of-the-box and custom features (principially in the form of workflows and timers) to make a number of our internal processes more consistent and efficient.
Sharepoint 2010′s increased mobile-friendliness allows the platform to detect mobile devices accessing its pages and redirect URLs to pages optimized for small screens and reduced bandwidth, and should provide easier access to collaboration tools and documents for our many staff members who are frequently offsite.
Finally, migration from our existing hosted 2007 Sharepoint solution is a great concern as we have used it extensively and have a fairly large amount of documents and other information stored there. Our hope is to perform the migration with the database attach upgrade feature of Sharepoint 2010, which (in theory) allows automatic upgrades of Sharepoint 2007 content databases to 2010 using commands in PowerShell.
