Talking Mapping at the 2014 ESIP Summer Meeting

Last week I had the opportunity to present at the Federation of Earth Science Information Partners (ESIP) Summer Meeting held in Copper Mountain, CO. The Summer Meeting is a gathering of IT professionals from across several different agencies such as NASA, NOAA and USGS. Each year, the group comes together to talk about the challenges that they each face while trying to engage and support the scientific community.

When I got in on Wednesday a few of us got together to talk about how to kickstart the Science on Drupal group. While there’s been a science presence in the Drupal community for several years now in one form or another, there’s been a recent interest in pooling resources together to make a larger group. We had a great time strategizing how to grow the group over chips and salsa.

For my presentation, I went over various different tools for doing online mapping work, both with native Drupal tools and other toolsets.

map of tonado

One of the big challenges that this community has to face is how to work with large datasets that don’t fit neatly into a typical Drupal site. For my part, we spent a lot of time going over how to leverage tools like D3, CartoDB, GeoServer, and Mapbox to connect to data outside of Drupal and provide meaningful interaction with it.

They also exposed me to DEIMS, a Drupal distribution that they had collaborated on that also features some interesting ways to interact with external data. There was a great presentation at Drupalcon Austin on the distribution that’s definitely worth checking out.

If you’re interested in catching the presentation, the slides are posted on Github and the video is here. If you’re interested in catching up with what’s going on with the Drupal in Science working group, check out their page on groups.drupal.org.

Thanks again to Adam Shepherd and the rest of the ESIP Drupal Working Group for inviting me out to hang out and learn from their experiences.

It’s Almost Time for Capital Camp and Drupal Gov Days!

We could not be more excited that two of our favorite DC events – Capital Camp and Drupal 4 Gov – are merging! The combined event, happening July 30th through August 1st at the National Institutes of Health, promises to be one of the most informative and inspiring conferences on Drupal and  open source in government yet. Phase2 is proud to be a platinum sponsor of what is sure to be an action-packed conference in the nation’s capital, our hometown.

Screen Shot 2014-07-09 at 1.28.42 PM

We’ve lined up 10 of our all-stars to present sessions at this year’s event (we told you we were excited!). Whether you’re interested in design, collaboration, or custom government solutions, we’ve got you covered. Here’s a sneak peek at our speaker roster…

content management solutions for government

Kick off your Capital Camp experience with a case study on How San Mateo County Is Raising the Bar with OpenPublic. Experience Director Shawn Mole and Program Director Felicia Haynes will discuss the technical challenges that San Mateo faced as a local government, and how they utilized Phase2’s Drupal distribution to overcome those obstacles. For more details on OpenPublic, catch OpenPublic 1.0: The Next Generation of Open Source Government Sites, presented by Shawn Mole and Greg Wilson, Director of Public Sector Practice at Phase2. Then learn how to create a “Sleep at Night CMS” with Senior Developer Randall Knutson.

 design and user experience

Necessary Capital Camp preparation: put these three sessions from Phase2’s front-end masterminds on your agenda. Start with Senior Developer Mason Wendell, who knows that great design, like jazz, needs both harmony and discord. His session, Thinking Inside the Box X3, will focus on component-driven design. Senior Designer Joey Groh will expose a real projects’ collaborative design process in his session, Collaborative Design to the Rescue: Photoshop in a post-Photoshop World. Finally, in his talk Amazing Design Through Empathy, Senior Experience Analyst David Spira will illustrate how to use empathy to improve all aspects of product design, from requirements gathering to user research and everything in between.

Mason3

 collaboration

Drupal has already proven to be a viable alternative to proprietary models for government CMS. Now Open Atrium is helping Drupal provide government agencies with an enterprise grade, open source platform to connect teams, departments and constituents. Learn from Greg Wilson and Mike Potter, Open Atrium’s Lead Architect, how OA2 addresses government collaboration needs in their talk, Open Source Collaboration for Government with Open Atrium. For a story of true open source collaboration and innovation, check out Director of Engineering Steven Merrill’s session on OpenShift and Drupal.

configuration, testing and site building

In recent years, Open Data has evolved from a buzzword to a reality to a requirement for governments, NPOs and NGOs globally. To explore what Open Data is, how to use it, and what it means to your organization’s website and its followers, stop by Senior Developer Robert Bates’ session, Open Data: Not Just a Buzzword. For more advanced developers, Steven Merrill will present on Open Source Logging and Metrics Tools, in which he will dive into the logging infrastructure of drupal.org and how you can apply the same tooling to your own sites. Finally, learn Best Practices for Development, Deployment, and Distributions from Mike Potter.

Be sure to visit our exhibitor booth to learn more about Phase2 and our people,  and of course  to grab some infamous Phase2 swag!  Are you attending Capital Camp and Drupal Days? What sessions are on your must-see list? Let us know below!

Open Atrium: The Open Source Enterprise Collaboration Solution for Government

In today’s digital world, government agencies are faced with the challenge of determining how best to connect not only internally, but externally with citizens and shareholders. At the federal, state, and municipal levels, agencies coordinate and share information with other agencies, with external communities, and across different levels of government.

Gov

As they attempt this coordination, government agencies must meet the collaboration needs of specialized projects, while operating within budget allocations and resource constraints.

“Challenge” may be an understatement. Luckily, a robust and secure toolset exists to successfully connect disparate government systems and constituents.

On Tuesday, June 24th we will be leading a discussion about collaboration in government and how Open Atrium can provide an open source enterprise solution to connect actors and engage citizens in the public sector.

oa

Open Atrium provides an open source, enterprise-grade collaboration platform for government that:

  • Engages constituents with a modern, mobile-friendly experience

  • Streamlines communication and workflows across groups

  • Integrates with enterprise systems

  • Provides the security that allows agencies to restrict access to information on both sides of the government firewall

Mike Potter, Open Atrium’s Lead Architect, and Greg Wilson, Director of Government Practice at Phase2, will discuss some of Open Atrium’s key features that make it a great fit for collaboration in government, including:

  • Secure document sharing and collaboration:  Securely keeps information in one place, unlike email.

  • Time management tools: Manage and monitor project activities with calendars and project tracking tools.

  • Security and access control: A robust access control system that outperforms any other open source solution.

  • Online communities and communication: Launching open or private discussions is simple.

  • Citizen and stakeholder engagement: Control the content the public sees.

Be sure to grab a seat at tomorrow’s webinar on Tuesday, June 24th at 12 PM EST to learn what Open Atrium can do for your agency!

 

San Mateo County- Raising The Bar For Local Government With OpenPublic

We recently celebrated the launch of San Mateo County‘s new and improved web platform using Drupal and OpenPublic. The San Mateo County digital team had a vision of a new web platform that would accommodate all departments with intuitive administrative functions as well as a well designed end user interface to efficiently deliver information to the County’s citizens. We new we could deliver this with our Drupal distribution OpenPublic.  OpenPublic is an open source distribution built with Drupal, developed to address commonly recurring challenges faced by Government agencies when managing their web content.  With the successful launch of the San Mateo County platform, I finally got the chance to sit down with Beverly Thames, Content and Collaboration Manager at San Mateo County to chat about the web platform redesign and OpenPublic, here is out Q&A:

Q: What was is the goal or mission of the San Mateo County digital presence and why did you need a redesign?

A: Between departments and central IT, the website was a multi-million dollar per year enterprise, and yet, County leaders were dissatisfied. The site failed to meet their needs or the needs of the public. Two goals of the redesign were to lower costs and provide better service to our departments and site visitors. We wanted to improve communications and to reduce or streamline in-person office visits.

We had major challenges with our old proprietary content management system’s inflexibility and highly technical nature. This made it difficult for departments to produce content and to organize that content in a meaningful way. The result was that much of the content grew stale while vital “evergreen” and fresh content was difficult for visitors to find via search or the site’s menus. Beyond these functional shortcomings, the frustrating user experience on the CMS was made worse by the outdated visual design, which displayed poorly in most modern browsers–especially when viewed on mobile devices. Overall, the sites did not communicate the San Mateo County or department brand very well, a situation compounded by the sites’ outdated content features. It was time for a change.

Q: What were the challenges and needs for San Mateo County’s digital presence and how did OpenPublic address them?

A: We chose to build our site on OpenPublic because it is tailored to the needs of government. County leadership could only be convinced to adopt open source if they were assured the system was secure and accessible. OpenPublic delivered both.

Each department’s identity and requirements are tied to their lines of business and the communities they serve. The County departments wanted to maintain their unique identities within the overall County brand. OpenPublic allowed the County to maintain a strong central brand while meeting user demand for autonomy and flexibility.

Q: Was using an open source platform an important factor in choosing a new content management system? If so, why?

A: San Mateo County was spending too much on annual licensing fees for a proprietary content management system that no one liked. There was little user support, and few developers knew the CMS, so when you could find one, their rates were high. Open source comes with a global community of support and many talented developers.

Open source is a natural solution for the government sector because we are constantly sharing our work with our peers and the public. Adopting an open source CMS allows the County to benefit from and contribute to the continuous improvement of the platform within the context of a larger user community.

Q: What is next for San Mateo’s digital presence?

A: Now that the site has been up for a few months, we will start digging into the analytics to see where we need to tweak search and to help us identify topics for curated pages.  We’re also excited about exploring potential integrations with our Open Data portal, GIS mapping and enterprise content management.

—————————

Learn more about San Mateo County’s new Drupal platform in our Portfolio, and get a deep dive look at how we developed an improved search functionality to better serve the County’s constituents.  If you are heading to DrupalCon Austin next week, don’t miss our session “Drupal For Gov- Raising The Bar With OpenPublic.”

 

Workflow within Open Atrium 2

atrium-logo (1)

A key requirement in most organizations is a content approval workflow.  The typical Drupal solution uses the Workbench Moderation module. However, the Workbench Moderation module only allows you to create a single site-wide workflow.  What if you need different workflows in different Open Atrium spaces, or between different content types?  The solution is the new Open Atrium Workbench module!

The Open Atrium Workbench module, along with several dependent modules, was a collaboration between Phase2 (srjosh and myself) and the Community (dsnopek).  It allows you to define multiple workflow “profiles” and apply them to content types within Open Atrium Spaces and Sections.  It also allows you to specify the Groups and/or Teams who are allowed to moderate content through the workflow.

Workbench Moderation Profiles

OA2_Workflow-3A workflow “profile” is a collection of States and Transitions.  A “State” represents where in the workflow a specific piece of content is, such as “Draft”, “Needs Review”, “Published”, etc.  A “Transition” is the act of moving between two states. Typically a Transition can only be made by a specific set of users with proper permissions.  For example, only members of a specific Working Group might be allowed to approve content, or only members of the Marketing team allowed to Publish content.

The Workbench Moderation module allows you to create these States and Transitions, but applies them globally across the entire site.  The new Workbench Moderation Profile module (currently a sandbox) creates a new entity-type for storing workflow profiles.  The States and Transitions are still created globally by Workbench Moderation, but the actual collection of these being used for a specific workflow are controlled by the profile entity.

Workbench Moderation Profiles is very generic and doesn’t care exactly how these profiles are applied.  Submodules to support Organic Groups and Content Types are provided.  Hooks are available for further control.  The Open Atrium Workbench module uses these hooks to provide Space-specific and Section-specific workflow profiles.

Turning it all on

To add workflow to Open Atrium, download and install the Open Atrium Workbench module and all it’s dependencies.  There are several patches needed to Workbench Moderation documented in the oa_workbench.make file.  Eventually these will get committed to Workbench Moderation and much of the Profile sandbox will be incorporated directly.

Once all of the modules are enabled, the various Drupal permissions and Organic Groups (OG) permissions need to be set.  Permissions in Workbench Moderation are a bit backwards from the norm:  submodules *revoke* access rather than *grant* access.  Typically this means you will set the Drupal Workbench Moderation permissions (including View Unpublished content) to be allowed for all authenticated users, then use OG permissions to restrict to Members and/or Space Admins, then optionally use OA permissions to restrict to Groups/Teams.

OA2 Workflow ContentAfter enabling permissions, you next need to enable Moderation of Revisions on the specific content types that you want to use workflow.  When using Open Atrium, enabling Moderation on a content type will not turn on the workflow features until the Space itself enables a Profile.  Typically this means you will enable Moderation on many content types, such as Document Pages, Events, etc.  In most cases, moderation will *not* be used on Discussion Posts since those are usually ad-hoc discussions that do not require approvals.

OA2_Workflow_Creation-2

Finally, the last step it to create your custom Workbench Profile containing the transitions you wish to include, then enable this workflow Profile for your Space or Section.  For Spaces, the workflow profile is set within the Config/Workbench Moderation page.  For Sections, the workflow profile is set by editing the Section node and setting the profile field.

To limit transitions to Groups or Teams, enable the oa_workbench_access module (included with oa_workbench).  To allow transitions to be scheduled automatically in the future, download and enable the Workbench Moderation Scheduled Transitions module (also a sandbox).

How it works

Once Open Atrium Workbench is configured, creating new content within a Section will display the normal Workbench Moderation messages panel.  This panel provides information about what State the document is in and allows options for moving to a different state if you are approved to make that transition.

OA2_Workflow_messages-3For example, a Member creates a new content Draft.  Once they are happy with the draft, they move it to the “Needs Review” state.  Somebody authorized to review the document visits their “My Workbench” page from their User Badge dropdown menu and goes to the “Needs Review” tab to see all of the content awaiting their approval.  After reviewing the document, they can either Reject the document, sending it back to the Draft state, or they can approve the document, sending it to the Published state.  Only the users authorized to publish the document will see the Published option in the workbench panel.

Once content is published, a new draft can still be made.  Workbench Moderation supports having one revision of content published while a different revision is in the draft state.  The new draft will only replace the currently published revision once it is approved and published via the workflow.

See you at DrupalCon Austin!

For more detailed information on using Open Atrium Workbench, watch my hands-on demo webinar.  If you are coming to DrupalCon Austin, stop by the Phase2 booth for a demo, or schedule a demo to discussion your specific organizational needs.  Or just come to our booth and say “Hi” and tell me about the cool and interesting ways you are using Open Atrium 2 in your organization.

While I’ve been using the default publishing approval workflow as my example, each organization has different workflow needs.  The workflow profile used for publishing documents is quite different from the workflow used to manage tasks or issues.  The workflow used in a private section (if any) is likely different than the workflow used in a public section.  Open Atrium supports all of these different cases in a systematic and easy-to-use way, consistent with users familiar with the Workbench Moderation module.  This functionality makes Open Atrium a key solution in the Enterprise Collaboration space on par with many non-Open-Source systems.

Combining Tasks with Grunt

I was recently asked to help out with a few build steps for a Drupal project using Grunt as its build system. The project’s Gruntfile.js has a drush:make task that utilizes the grunt-drush package to run Drush make. This task in included in a file under the tasks directory in the main repository.

tasks/drush.js

You can see that the task contains a few instances of variable interpolation, such as <%= config.srcPaths.make %>. By convention, the values of these variables go in a file called Gruntconfig.json and are set using the grunt.initConfigmethod. In addition, the configuration for the default task lives in a file called Gruntfile.js. I have put trimmed examples of each below.

Gruntfile.js

Gruntconfig.json

As you can see, the project’s Gruntfile.js also has a clean:default task to remove the built site and a mkdir:inittask to make the build/html directory, and the three tasks are combined with grunt.registerTask to make the default task which will be run when you invoke grunt with no arguments.

A small change

In Phase2′s build setup using Phing we have a task that will run drush make when the Makefile’s modified time is newer than the built site. This allows a user to invoke the build tool and only spend the time doing a drush make if the Makefile has indeed changed.

The setup needed to do this in Phing is configured in XML: if an index.php file exists and it is newer than the Makefile, don’t run drush make. Otherwise, delete the built site and run drush make. The necessary configuration to do this in a Phing build.xml is below.

build.xml

You’ll note that Phing also uses variable interpolation. The syntax, ${html}, is similar to regular PHP string interpolation. By convention, parameters for a Phing build live in a build.properties file.

A newer grunt

The grunt-newer plugin appears to be the proper way to handle this. It creates a new task prefixed with newer: to any other defined tasks. If your task has a src and dest parameter, it will check that src is newer than dest before running the task.

In my first quick testing, I added a spurious src parameter to the drush:make task and then invoked the newer:drush:maketask.

That modification worked properly in concert with grunt-newer (and the drush task from grunt-drush task didn’t complain about the extra src parameter,) but I still also needed to conditionally run the clean:default and mkdir:init only if the Makefile was newer than the built site.

Synchronized grunting

The answer turned out to be to create a composite task using grunt.registerTask and grunt.task.run that combined the three tasks existing tasks and then use the grunt-newerversion of that task. The solution looked much like the following.

tasks/drushmake.js

I could then invoke newer:drushmake:default in my Gruntfile.js and only delete and rebuild the site when there were changes to the Makefile.

Learn more about build systems in Adam Ross’s blog post “Creating Options in Automated Software Deployment.”

Our Top 10 DrupalCon Austin Session Picks!

It’s less than 2 weeks from DrupalCon Austin and we are pumped to check out this year’s awesome DrupalCon Austin session lineup. With so many sessions to choose from, it can be hard to pick which ones to attend, so planning your session strategy is essential. To help you decide, we’ve put together our top 10 DrupalCon Austin session picks below, we’ll see you there!

Monday

9:00-5:00PM:  Drupal Performance and Scalability with the Dream Team.

The dream team unites again to provide an epic performance and scaling training to kick off this year’s DrupalCon.  You’ll find our own DevOps guru, Steven Merrill among this infamous team and while this isn’t a DrupalCon session, it is definitely a training not to be missed. Try to grab a seat!

Tuesday

1:00-2:00PM: 30 Drupal 8 API Functions You Should Already Know, Rm: 18, 4th floor

You wont want to miss one of Phase2′s leading developers, Frederic Mitchell presenting a session on Drupal 8 API functions.  You can get a sneak peak of what his session will cover in his blogpost “Say Goodbye To menu_get_object().”

1:00-2:00PM: OpenShift & Drupal- A True Story of Open Source Collaboration, Rm 11,4th floor

Have you heard about using OpenShift Origin with Drupal? Get all the details and find out how to easily deploy Drupal sites on OpenShift Origin in this session with Steven Merrill and Diane Mueller.

3:45-4:45: Introducing The Drupal 8 Configuration System, Rm F, 4th Floor

Drupal 8 has a new configuration management system that provides a central place for modules to store configuration data, from simple static data (e.g., site name) to more complex business objects (e.g., field definitions and image styles). This session will outline how to work with the new configuration system from a site building perspective. Looks like a great D8 session, we can’t wait!

5:00-6:00PM: Train Wrecks, Ugly Babies- The Gory Details Part 2, Rm 16, 4th Floor

As a follow up to Susan Rust‘s previous session “Train Wrecks, Ugly-Baby..Meetings and other Client Calamities,” Susan further discusses project methodology and the importance of discovery and strategic planning.  We are a big fan of Susan and look forward to her session!

Wednesday

1:00-2:00: Introducing Panopoly and Drupal Distributions, Rm 17, 4th Floor

If you are interested in how Panopoly works and how it is used in Drupal distributions, be sure to catch David Snopek‘s session! David is a fantastic Open Atrium and Panopoly contrbutor and has a ton of knowledge under his belt about both distributions!

3:45-4:45PM: Successful Requirements Gathering, Rm 16, 4th floor

Jordan Hirsch is a senior digital strategist here at Phase2.  He is not only a professional improv actor, he also is an expert in requirements gathering for large complicated web projects.  Check out his session at DrupalCon and take a sneak peak at some of his requirements best practices in his blogpost about this topic.

3:45-4:45PM: Thinking Inside The Box, Inside The Box, Inside The Box,  Rm F, 4th floor

For all you frontend developers out there, be sure to catch Mason Wendell‘s session where he will be discussing how to leverage style prototypes as a frontend style guide technique.

Thursday:

10:45-11:45: Scaling Drupal For Higher-Ed Institutions, Rm 16, 4th Floor

For those interested in Drupal for higher ed, be sure to catch this session, we are really excited to attend this session with their incredible line up of representatives from Dartmouth College, Harvard University, Stanford University, and Yale University.

2:15-3:15PM: Drupal For Local Gov- How San Mateo raised the bar with OpenPublic, Rm 17, 4th floor

If you are interested in Drupal for public sector sites, you won’t want to miss this session.  Join Felicia Hayes of Phase2 and Beverly Thames from San Mateo County to talk about how San Mateo County leveraged Drupal and OpenPublic to create an engaging web experience. will present a session on the San Mateo project with Beverly Thames of San Mateo.

With all these interesting Drupal sessions to attend, make sure to pace yourself, take a break and visit us at the Phase2 booth (#101)! As always we’ve got some fun swag and activities planned at our booth, so be sure to schedule some time to come say hi!

Creating Custom Panels Layouts & Content Types

The Panels module provides a flexible and intuitive way to place content throughout your site. This blog post will show you how to use and customize some of the features in panels including layouts and content types.

Layouts

As it sounds, layouts allow users to choose where each piece of content on a given panel can be rendered. There are two different types of layouts. Builders allow users to create a custom layout inside the GUI via the layout Designer, while predefined layouts allow you to write the actual html to be used within your layout. This blog post will cover the latter option.

When creating a layout you can create one either inside your theme or within your module. For this example we will be creating a two-column layout using the theme implementation. In your theme create a folder named “layouts” and in your theme_name.info file add the following lines.

This line tells the theme to look for custom layouts within our layouts directory inside our theme. Inside of your Layout’s folder, create a folder called one_third_left and inside of that folder create three files, one_third_left.inc, one-third-left.tpl.php, one-third-left.css. Your directory structure should be as follows:

 

The .inc file controls the regions that are present as well as the theme and css to be used within the layout. Open the one_third_left.inc file and add the following lines.

This array defines all the needed items for a layout to appear. The title as it sounds, is the title of the panel we created. Category is used to group similar layouts together. When choosing a layout you first need to select the category and then the corresponding layouts will be shown. The icon is a visible representation to help users see what the layout looks like at a glance (The icon was omitted for this tutorial). The theme is the tpl file used to control the layout and html. Css controls the styling of the layout, it is also important to know that the css file is also used to control the layout shown inside the gui. Regions are used to define where content can be placed inside of the layout.

Open one-third-left.tpl.php and add the following lines.

In this tpl example, we are creating a 2 column layout with a header. Notice the $content variable. Each of these corresponds to a region that we created inside of our .inc file. To add more regions simply add them to the .inc file and place them in the tpl. I will not cover the css in this tutorial but once cache is cleared you should now see your new layout available for use.

Content Types

Ctools content types are like blocks in the sense that they both provide an easy way to create pieces of content that can be placed throughout a site. Where content types excel is in the configuration. While blocks allow you to configure them by means of a form, those settings are global, so every instance of a block will have the same settings no matter where it is placed. Content Types on the other hand allow you to create one content type and have different settings applied. This portion of the tutorial will discuss how a ctools content type can be created.

First add ctools as a dependency in your module_name.info file.

Create a folder named “plugins” inside of your module directory. Next add a folder named “content_types” inside of the newly created plugins directory. Then add the following lines to your module_name.module file.

This hook tells ctools where to look for its content types. In this example we will create a simple content type that displays the 5 newest published nodes and allows the user to choose whether or not the title is linkable.

Inside the content_type folder, create another folder called newest_node and inside that folder create a file called newest_nodes.inc. Add the following lines to the file.

This array is used to define our settings for the content type. As it sounds, title is the title of the content type. Single is used to show that our content type has no subtypes. The render callback is the function that is used to generate the markup for our content type. Defaults is the default ctools context. In our case we do not have one. The edit form is the form function used to generate the content type settings. Finally category is used to group similar content types. To add the form add the following lines.

This simple form let users decide how many nodes to display and whether or not a link should be displayed. Now lets proceed with adding the actual render function.

This function does a simple db_select using the variables we set in our form to limit the number of items displayed and to also decide whether or not to display a link. Clear your cache and you should now see your newly created content type available along with the settings form.

 

Finally here is what my Content Type looks like once styled.

While my previous example is a bit on the simplistic side it does highlight the power of ctools layouts and content types. When used with the appropriate CSS responsive layouts can be created and easily changed per node type. For more information on the Panels Module, check out some of our other Phase2 thoughts.

Powered By Open Atrium, Run On OpenShift.

At the Red Hat Summit this April, we were thrilled to announce the launch of the  OpenShift Origin community site using Open Atrium in partnership with OpenShift. Open Atrium is a social collaboration Drupal distribution architected and maintained by Phase2. As the first Drupal site to launch on OpenShift Origin, we were excited to develop this proof of concept and lead the way for Drupal development using OpenShift Origin as a Platform as a Service(PaaS) application. I recently caught up with Diane Mueller, OpenShift Origin’s Community Manager, to get the scoop on how they are using Open Atrium to promote collaboration within the OpenShift community.

Q: What is the mission of the OpenShift Origin Site and what role does collaboration play?

A: OpenShift has multiple communities of users ranging from PaaS administrators who need to connect and collaborate on the best practices for deploying and managing OpenShift; developers building applications that leverage OpenShift’s architectural model; cartridge builders who are extending the OpenShift ecosystem with new pluggable cartridges for a myriad of languages, frameworks, databases and other services; and hosting providers deploying OpenShift on their clouds as a public offerings. Each of these unique communities bring different perspectives, different use cases, and different levels of expertise. As you can expect, there’s a lot of cross-pollination and collaboration that goes on in this complex ecosystem of an open source project which helps inform the project’s direction and shape our community’s road map.

Q: What were the challenges and needs for the OpenShift Origin site? Why did you choose Open Atrium to address them?

A: Open Atrium gives us room to grow. It has a robust functionality and a project management aspect that a vanilla CMS just couldn’t provide for us. As a technology-oriented community, it’s essential that we have the ability to collaborate effectively and efficiently.

Q: Was using an open source platform an important factor in choosing a website software? If so why?

A: Well, we’re Red Hat so, of course, open source is important to us. We sponsor over 100,000+ open source projects and OpenShift is just one of them. As the community manager, it was important that we walk the talk. I believe it’s important to have collaboration across open source communities in order to move code forward. We’re happy to be working with the Drupal and Open Atrium communities and to be driving OpenShift to be a first-class host for all flavors of Drupal.

Q:What is the most useful feature in Open Atrium for your collaboration needs?

A: I have been finding the “Sections” metaphor most useful when dividing up responsibilities amongst the various content creators for the Origin site. We’re able to easily give control over to different content editors and creators. They, in turn, are empowered to have more creative freedom. As the webmaster of the site, I know that their changes won’t interfere with the rest of the site.

Q: How do you see your site evolving or maturing going forward with Open Atrium?

A: We’ll be moving the full technical documentation set over to Open Atrium soon and bring that aspect of the project more front and center on the site. We’re also reaching out to our global community. We hope to have the Latin America and EMEA content sections integrated in the near future.

——————————————

This is all very exciting Diane! Thanks so much for catching up with us on all the OpenShift Origin community development powered by Open Atrium.  We’re are thrilled to be OpenShift partners and to introduce the OpenShift Origin community to Open Atrium. We look forward to seeing how this project grows and scales as a hub for open source collaboration!  Are you heading to DrupalCon Austin? Don’t miss Diane Mueller and our own Steven Merrill presenting a session on OpenShift and Drupal! Meanwhile, learn more about Phase2’s partnership and open source collaboration work with OpenShift.

 

Better, Stronger, Faster! A New Open Atrium Installer.

Open AtriumInstalling Drupal from scratch is a relatively painless for most people.  However, installing a large Drupal distribution such as Open Atrium 2 has the potential to be painful.  Core Drupal only contains a handful of modules that need to be installed, but Open Atrium 2 is a very feature-rich distribution with nearly 200 modules to be installed.  An installation of Open Atrium typically takes 15-20 minutes and consumes significant server resources during that period.  In this article, I will show how we reduced that installation time down to only TWO minutes!

Overview of Drupal Installation

Have you ever looked at the Drupal core installer code, or traced through it?  It’s actually a fairly robust and flexible installer framework.  The installation consists of several different “steps”.  Each step can be a form asking for information, such as the database information, site information, theme selection, etc.  Or a step can simply be a function that performs processing, such as the step to check if the system requirements of Drupal are met.  Each step can optionally be processed via the Batch API, such as the step that installs each required module.

The Drupal installer can be run interactively, or non-interactively, such as when using the “drush site-install” command.  During interactive installs, separate HTTP requests are used for each step of the process, and for each “batch” of processing done by the batch steps.  Batch processing helps avoid timeout errors, but a single step that takes too long can still cause a timeout.  The installer tries to break batch steps into one-second requests, but that only allows fast steps to be combined into single requests.  A batch step that takes more than one second only ensures that a new request is used for the next batch, and can still cause a timeout.

Caching during the installation process is also very complex.  In general the installer minimizes the amount of cache clearing to speed up the install process.  However, during module installation, some modules might depend on the existence of other modules and without some cache clearing, modules installed during the same batch request might not see each other.  This becomes even more difficult if a module is actually a Features export.  Features has it’s own caches and sometimes needs to rebuild a feature to put configuration into the database that might be needed by other modules.

Installing Open Atrium 2

What appears to be a straight-forward installation process for Drupal core becomes exponentially more complicated when installing a distribution that uses hundreds of modules and Features, such as Open Atrium.  An interactive installation of Atrium begins fine and quickly enables the first 30 modules.  Soon it becomes slower and slower.  Enabling the last module takes nearly ten times the first module.  Each Feature module that is installed causes the info files of all previously enabled modules to be reloaded into cache.   So the more modules installed, the slower Features becomes.  There are many issues, such as this in the Features queue to try and address some of these performance issues, but it is a very complex process.

Because modules take longer and longer to enable, some Open Atrium installs run into PHP timeout problems and need to increase their limit from the 30-second default up to 60-seconds.  This is often frustrating because these timeouts often near the end of the installer after 15 minutes have already passed.  In general, a successful Open Atrium 2 installation takes around 15 to 20 minutes, which is far too long for most people trying to make a quick evaluation of Open Atrium functionality.

There Must be a Better Way!

During the recent Drupal NYC Camp, we held an Open Atrium 2 training session.  During this training, we had 20 people all installing Open Atrium at the same time.  We all collectively waited 20 minutes just to get 20 identical copies of Open Atrium 2 installed on our computers.  Wouldn’t it have been nice to simply install Open Atrium once and then clone that onto the other computers?  That’s when I realized there was a better way to install Open Atrium…by cloning a previous installation!

OA2_Install_Option-2

The new 2.18 version of Open Atrium contains this new installer option.  After entering your database information, you are prompted for the Installation Method.  The Standard Drupal installation method is still available (if you really want to wait 20 minutes!).  The new Quick installation option is the default.  Selecting Quick installation will direct the installer to import from an existing database dump that is saved within the Open Atrium code repository.

Instead of a batch process step to install and enable each module, the Quick install step uses a batch process to import database tables from the sql dump.  This will only work if you are using MySQL.  If you are using a different database engine you won’t get prompted for the new Quick install…it will just use the Drupal default installer.

After only TWO minutes, the installer should finish importing the database tables.  You will then be prompted to fill in your Site information as normal.

Technical Details

Importing an existing database from the Drupal installer turned out to be trickier than originally anticipated.  The Drupal installer isn’t happy when certain database tables or variables stored in the database are ripped out from under it.  Another complexity was the fact that the new database might have a different database “prefix” for table names.  The new installer code actually parses the MySQL database dump line by line to split the SQL statements into batches for each table and to replace table names with the new database prefix.

To visualize the difference between the standard Drupal installer and the new Quick install, I ran both and sent the statistics into Graphite:

OA2_Install_Graph-2

The first 20 minutes of this graph shows the normal Drupal installation of Open Atrium 2.  The CPU is almost fully utilized during the entire install process.  The resident memory used by Apache climbs as the installer adds more and more modules to the site.  At around 08:27 the first Feature export module is enabled, causing a sharp increase in the amount of memory being used.  The decrease in CPU at 08:30.5 (when the green CPU line goes to zero) is caused by the installer asking for Site Information.  After that step, the Drupal installer clears all caches, runs cron, and does other cleanup.  This increases the resident Apache memory even further.

The second set of data starting at 08:36 is for the new Quick install option.  You’ll notice the actual install time is around 2-3 minutes.  Once again, the decrease in CPU at 08:38.5 is when the installer prompted for Site information.  Thus, the data after that represents the same cache clearing and install cleanup as in the full installer.  Ultimately the same amount of memory is needed to fully clear the cache and clean up the site as the same modules have been installed and enabled at that point.

In fact, the Quick installation process itself consists of importing over 300 database tables, followed by it’s own cache clear.  The memory increase at around 08:37.5 is caused by cache clear that the Quick installer executes after all the database tables have been imported.  During the actual database import, memory usage in Apache is flat.

Conclusion

Any other Drupal distribution that requires a large number of modules should be able to leverage the code used in Open Atrium.  The code is all within the “install_from_db” subdirectory of the profile, which can be found in the Open Atrium project on drupal.org.  The huge reduction in installation time should help retain new clients that are evaluating Open Atrium and Drupal for their organizations and generally improve first impressions of Drupal.