Simplify Your Logstash Configuration

As I mentioned in my recent post, I got a chance to upgrade the drupal.org ELK stack last week. In doing so, I got to take a look at a Logstash configuration that I created over a year ago, and in the course of doing so, clean up some less-than-optimal configurations based on a year worth of experience and simplify the configuration file a great deal.

The Drupal.org Logging Setup

Drupal.org is served by a large (and growing) number of servers. They all ship their logs to a central logging server for archival, and around a month’s worth are kept in the ELK stack for analysis.

Logs for Varnish, Apache, and syslog are forwarded to a centralized log server for analysis by Logstash. Drupal messages are output to syslog using Drupal core’s syslog module so that logging does not add writes to Drupal.org’s busy database servers. (@TODO: Check if these paths can be published.) Apache logs end up in/var/log/apache_logs/$MACHINE/$VHOST/transfer/$DATE.log, Varnish logs end up in/var/log/varnish_logs/$MACHINE/varnishncsa-$DATE.log and syslog logs end up in /var/log/HOSTS/$MACHINE/$DATE.log. All types of logs get gzipped 1 day after they are closed to save disk space.

Pulling Contextual Smarts From Logs

The Varnish and Apache logs do not contain any content in the logfiles to identify which machine they are from, but the file input sets a path field that can be matched with grok to pull out the machine name from the path and put it into the logsource field, which Grok’s SYSLOGLINE pattern will set when analyzing syslog logs.

Filtering on the logsource field can be quite helpful in the Kibana web UI if a single machine is suspected of behaving weirdly.

Using Grok Overwrite

Consider this snippet from the original version of the Varnish configuration. As I mentioned in my presentation, Varnish logs are nice in that they inclue the HTTP Host header so that you can see exactly which hostname or IP was requested. This makes sense for a daemon like Varnish which does not necessarily have a native concept of virtual hosts (vhosts,) whereas nginx and Apache default to logging by vhost.

Each Logstash configuration snippet shown below assumes that Apache and Varnish logs have already been processed using theCOMBINEDAPACHELOG grok pattern, like so.

The following snippet was used to normalize Varnish’s request headers to not include https?:// and the Host header so that therequest field in Apache and Varnish logs will be exactly the same and any filtering of web logs can be performed with the vhost andlogsource fields.

As written, this snippet copies the request field into a new field called full_request and then unsets the original request field and then uses a grok filter to parse both the vhost and request fields out of that synthesized full_request field. Finally, it deletesfull_request.

The original approach works, but it takes a number of step and mutations to work. The grok filter has a parameter calledoverwrite that allows this configuration stanza to be considerably simlified. The overwrite paramter accepts an array of values thatgrok should overwrite if it finds matches. By using overwrite, I was able to remove all of the mutate filters from my configuration, and the enture thing now looks like the following.

Much simpler, isn’t it? 2 grok filters and 3 mutate filters have been combined into a single grok filter with two matching patterns and a single field that it can overwrite. Also note that this version of the configuration passes a hash into the grok filter. Every example I’ve seen just passes an array to grok, but the documentation for the grok filter states that it takes a hash, and this works fine.

Ensuring Field Types

Recent versions of Kibana have also gotten the useful ability to do statistics calculations on the current working dataset. So for example, you can have Kibana display the mean number of bytes sent or the standard deviation of backend response times (if you are capturing them – see my DrupalCon Amsterdam slides for more information on how to do this and how to normalize it between Apache, nginx, and Varnish.) Then, if you filter down to all requests for a single vhost or a set of paths, the statistics will update.

Kibana will only show this option for numerical fields, however, and by default any data that has been parsed with a grok filter will be a string. Converting string fields to other types is a much better use of the mutate filter. Here is an example of converting the bytes and the response code to integers using a mutate filer.

Lessons Learned

Logstash is a very powerful tool, and small things like the grok overwrite parameter and the mutate convert parameter can help make your log processing configuration simpler and result in more usefulness out of your ELK cluster. Check out Chris Johnson’s post about adding MySQL Slow Query Logs to Logstash!

If you have any other useful Logstash tips and tricks, leave them in the comments!

 

DrupalCon Amsterdam Roundup

As I write this, I’m on a plane back to the US after a whirlwind 10 days in Amsterdam for DrupalCon Amsterdam 2014. As always it is so gratifying to meet and work with the international Drupal community. I love getting to take a look at what everyone is working on and collaborate with people that you might otherwise only know as IRC nicks. Here are my highlights of DrupalCon Amsterdam:

Sprints and Drupal.org Logging

I volunteer my time to help the drupal.org infrastructure team with their logging infrastructure. I was very happy to be able to sprint for 3 days, mainly on drupal.org infrastructure. On Sunday, Friday, and Saturday, I worked with the Drupal.org Puppet configuration to get a new CentOS 6-based log aggregation host ready to go running the latest versions of the ELK (ElasticSearch, Logstash, and Kibana) stack. The new Logstash configuration that we’ll be rolling out is much simpler. Stay tuned to the blog for information on some of the improvements we made in the process. We hope to deploy the new logging host within the week.

DevOps Meetup

Tuesday evening, I attended the DevOps Amsterdam meetup.  The meetup was sponsored by ElasticSearch, who bought a delicious dinner for all attendees as well as some drinks. During dinner, I sat with some folks from Germany and had a chance to speak with a number of ops-attuned folks about Docker and the possible use cases for it.

The meetup had some great content. There was a talk on how GitLab uses omnibus to package GitLab with far less hassle, a talk from fellow d.o infra team volunteer Ricardo Amaro on building the next-gen Drupal testbot on Docker, and a great talk from the DrupalCon Amsterdam DevOps chair Bastian Widmer on developing culture and sharing knowledge in an agency.

LSD Leadership Meeting

Earlier this year, Phase2 contributed some of its innovation hours to the LSD project and worked with Acquia to present a webinar on Behat and release a pre-built virtual machine designed to make it easy to start doing automated testing using Behat. During the LSD leadership meeting I joined Melissa Anderson of Tag1 Consulting and Hugh Fidgens of Pfizer in a breakout session discussing Behat. Quite a few organizations present were very interested in how they could use Behat to enable a behavior-driven development workflow with their developers, or to develop a good set of automated tests that could be run either as smoke tests or as end-to-end integration tests.

Behat Everywhere

In addition to the LSD leadership meeting, automated testing and BDD were topics on everyone’s minds throughout DrupalCon.

I attended 2 BoF about automated testing or Behat, the Testing Drupal 8 BoF, and Hugh Fidgen’s Organizational Behat BoF. These talked about how we could better leverage automated testing tools in Drupal core and in client work we build today, respectively. Many people have had some success in getting automated tests or a BDD workflow started, but there was a lot of talk about writing good sustainable tests and how to integrate these tools into your workflow.

Speaking of writing sustainable tests, my favorite session of the conference was definitely the session by Konstantin Kudryashov (the creator of Behat and Mink) on how to do BDD with Behat. The session was remarkable and left an impact on many folks who went there. It really emphasized the point that BDD must be about identifying and delivering business value in our projects, and that doing that is the way to write good tests. It is definitely worth an hour of your time to watch.

As testing best practices are refined in the Drupal community as well as in Phase2, I’m very excited that the talented Jonathan Hedstrom has joined the Phase2 team.  Jonathan is a maintainer of the Behat DrupalExtension, and is sure to help us further refine our testing practices as Phase2. Jonathan has been doing work recently on upgrading the DrupalExtension to support Behat 3 and has plans to generalize the drivers that the Behat DrupalExtensions provides so that it could be used with Mink for writing tests in straight PHP without needing to use Behat.

My “Open Source Logging and Monitoring Tools” Session

On Wednesday I presented my session “Using Open Source Logging and Monitoring Tools.” I covered a lot of information in my session including using Logstash, ElasticSearch, and Kibana.  Thanks to the tireless work of the DrupalCon A/C crew, the video recording of my session is online, and the slides are available on SlideShare if you would like to follow along.  The session had an excellent turnout, and there were some great questions and discussions following my session. I was quite pleased at the large turnout, and so was @KrisBuytaert, who been bringing information about DevOps-related topics to DrupalCons for years.

 

 

We were also fortunate to have Leslie Hawthorne of ElasticSearch in the audience. She gave out ElasticSearch ELKs to sprinters at the Friday event.

 

Takeaways

Based on the sheer number of people interested and sessions to devoted to both topics, it is clear that there is a growing interest in both logging and metrics and automated testing or BDD in the Drupal Community. This is also the 11th DrupalCon I’ve been to, and this year’s keynotes were the best I could remember. I really enjoyed Dries’s ideation around sustainable methods for getting contributions to Drupal core and contrib, and getting to see Cory Doctorow speak live about the perils of DRM and restrictions software freedoms was also excellent.

This is an exciting time for Drupal. Drupal 8 beta 1 is live. The community is active and engaged around making Drupal better, both by contributing to Drupal 8 and by doing a better job testing projects built on it. The DA and an army of volunteers have made huge strides on improving the infrastructure around core testing as well as around all the online communities under drupal.org.

Introducing DrupalCon Amsterdam Hackathon Champs!

Last Monday, Phase2 hosted a distribution hackathon at DrupalCon Amsterdam. While we were excited to see what our hackers would come up with, we were blown away by what was accomplished in an evening of hacking.  The winners of the hackathon developed a Drupal 8 distribution called Drupal Promo Kit. This distribution allows anyone to easily create presentations and landing pages using Drupal. Other hack projects included Open Atrium Apps, Panopoly apps, and more. I got to chat with the hackathon winners: Kate Marshalkina , Konstantin Komelin, John Ennew, and Mariano Barcia, to learn how they came up with their idea and how they did it:

Hackathon Winners

Q: What was your inspiration for your hackathon project?

A: We wanted to test our skills in Drupal 8 and see if we could build our own distribution.  We also wanted to build a unique solution that was useful and meaningful.

Q: How did you prep for the hackathon?

A: About a week before the hackathon Kate and I (Konstantin) brainstormed potential hackathon ideas. Our goal was to decide on a project that we could complete by the end of the hackathon, so once we decided on our idea for the project, we decided on the scope of our project and the minimal and maximum viable product.  We then decided on the tools we would use including Bitbucket, Google docs, Slack, and Trello.

Q: And once you arrived at the hackathon, what was your experience working with other people?

A: When the hackathon kicked off, we announced our project and goals to everyone and found some new team members that were interested in what we wanted to do. John Ennew and Mariano Barcia joined us and it was amazing how easy it was to pick up and start developing with other Drupalers. We were able to start developing rapidly in just a couple of hours. It didn’t matter that we were all coming from different backgrounds, with different skill levels, we were all speaking Drupal, a universal language.

Q: What do you think is the value of hackathons?

A: Hackathons are a great opportunity to meet other Drupalers in the greater Drupal community and work together to solve problems. Hackathons at Drupalcon are special because your  team can come from all over the world, and you all have different cultures, different jokes, and new ideas.  Not only do you build friendships throughout the event, but all the different perspectives and experiences strengthen your project. Hackathons let you dive into something you are interested in and at the same time, it can help push Drupal forward.  We learned so much about Drupal 8 while working on this project, and we plan on taking our experience and feedback to the core team to help improve Drupal 8.

—————————-

Interested in learning more about Drupal Promo Kit? Check it out on Drupal.org! We want to thank all the hackers that participated in our hackathon, and can’t wait to hack with our brilliant community again soon!  See more images of the hackathon and other DrupalCon Amsterdam photos on our Flickr!

By4dnAPCAAA-qSo

Introducing OpenPublic 1.0: The Next Era of Digital Government

Since 2011, OpenPublic has been transforming government by building government websites and applications which ensure security, mobility, and accessibility. Through our work with numerous government agencies, our teams have developed deep expertise at recognizing and responding to the specific technical challenges faced by public sector organizations.

But that doesn’t mean we couldn’t get better – and we did. As of today, we are proud to introduce the new and improved OpenPublic 1.0. It is the culmination of years of developing content management platforms for federal, state, and local government agencies. With each project, we gained a better understanding of these organizations’ digital needs. From the Department of Homeland Security, to Georgia.gov, to the recent launch of San Mateo County’s multi-site platform, OpenPublic has evolved to the mature product it is today.

What’s New in OpenPublic 1.0?

The 1.0 product version encapsulates all the most important OpenPublic functionality in a clean collection of Apps, simplifying the distribution’s powerful out-of-the-box capabilities. Not only does OpenPublic 1.0 break the mold by “appifying” what was once a wilderness of modules and complicated Drupal configuration, the newly released product is also fully compliant with the Federal Information Security Management Act (FISMA).

The conception of OpenPublic 1.0 was based on Phase2’s significant experience building government technology solutions. While working with San Mateo County, we developed the idea of using apps to make the distribution’s functionality simple to configure for site administrators. Instead of wading through Drupal’s confusing configuration settings and modules, admins can now turn features on and off without affecting other parts of the platform. Apps like the Services App, Media Room App, Security App, and Workflow App provide distinct segments of functionality specifically designed to complement agencies’ digital needs. In July, Experience Director Shawn Mole elaborated on our OpenPublic App strategy and its potential to transform content management for the public sector.

Screen Shot 2014-09-24 at 2.08.42 PM

Open Technology and Government

Like all distributions maintained by Phase2, OpenPublic is built with open technology, with good reason. Government agencies strive to reduce unnecessary costs for their taxpayers, and avoid the recurring licensing fees of proprietary software is a major benefit to open source solutions. Bypassing proprietary vendor lock-ins allows government to leverage the sustainable innovation of an open community working collaboratively to create, improve, and extend functionality, in addition to utilizing the community’s best practices for development. And because open technology is in the public domain, any agency can download, test drive, and learn about potential content management systems before choosing a provider.

sm5

San Mateo County, which worked with Phase2 to implement OpenPublic for the county’s CMS, recognized the value of openness in government technology and opened their code to the GitHub community. We were ecstatic that one of our clients embraced the open practices which are not only inherent in our work but laid the foundation for the development of OpenPublic. By making the innovative technology that went into building San Mateo County’s platform available for wider use, San Mateo County contributed to the government’s objective to foster openness, lower costs, and enhance service delivery. The “Open San Mateo” project demonstrates the power of open source to improve not just one government agency, but hundreds simultaneously by making the code available to other governments.

OpenPublic Moving Forward

We are hopeful that OpenPublic 1.0 will continue to advance more open government initiatives. In the meantime, Phase2 has several exciting projects in which we’ll show off some of the product’s enhanced features. Keep an eye open for the launch of the Department of the Interior (among others)!

Learn more about OpenPublic 1.0 at openpublicapp.com. For more information about Phase2’s services and how we can help build your site, email us at openpublic@phase2technology.com or comment below!

Phase2 and OpenSourcery: What’s New in Portland?

Once again, Phase2 has some exciting developments happening in Portland. Earlier this year, we announced the expansion of our team with the addition of the talented team from SuperStar Media. Now, we are proud to announce that we are expanding again with the amazing addition of the “superstars” of the OpenSourcery team.

This month, we have hired 14 members of this great company, which is shutting down operations on October 1. Due to their active participation in the Drupal and open source communities over the past 10 years, we knew the OpenSourcery team to not only be immensely skilled and experienced, but also very well-matched with Phase2’s collaborative and fun culture.

OLYMPUS DIGITAL CAMERA

So how did we find ourselves in the position to add these 14 exceptional employees to the Phase2 family? The close-knit, collaborative nature of the open source communities (particularly in Portland!) played a big role. As two experienced Drupal shops, Phase2 and OpenSourcery have maintained a presence at many of the same events, camps, and cons throughout the years, but connections between the two firms existed on a personal level as well. Many of our Portland-based staff regularly attended the city’s Drupal User Group, which has been hosted by OpenSourcery for the past six years. We engaged the team to work on our projects and found that indeed their talent and values were well aligned to our own.

OLYMPUS DIGITAL CAMERA

Luckily, this opportunity with OpenSourcery came at precisely the right moment for Phase2 in terms of our West Coast expansion. When we opened our initial Portland office in January, we saw enormous potential for growth in Portland, a city that is a hub of technical innovation and home to some of the most talented people working in open source. However, we could not have predicted how quickly our team would take root and flourish from one to thirty just this year!

As Project Manager Amye Scavarda wrote in a recent blog post, “Like the city itself, Phase2 embraces growth, innovation, partnership – and above all, culture.” I’d take this statement one step farther, and say that not only Portland but the open source community as a whole stands for these values. We are truly lucky to have found more people here who understand the open principles and shared values that drive our company culture.

OLYMPUS DIGITAL CAMERA

So, I am proud and excited to be able to introduce the 14 newest members of the Phase2 family! Today, we welcome Anthony Cast, Jonathan HedstromMaria Fisher, Jesse Hofmann-Smith, Delona Lang, Mike Nielson, Eric Paul, Austin Prohl, Anne Sturdivant, Jeff Wax, Sheena Morris, Chris Wright, John Cartier, and Ian Chamberlain. Welcome aboard!

OLYMPUS DIGITAL CAMERA

Our new PDX home!

 

Better Development Through Emotional Intelligence

I am unabashedly an engineer. I obsess over the pursuit of finding the most efficient solution to any problem.

In the realm of open source software, this approach has served me well. We read, reverse-engineer, fork, improve, and share. I want my process to be faster, more flexible, and maintainable for the long haul.

As I’ve investigated different methodologies, one characteristic I constantly underestimate is the team dynamic. I tend to pigeon-hole my mind into thinking that the solution to a problem is the most important goal.

Hey look! There’s a problem! I must find a solution for it.

  • What if complex problems can’t be solved by me?
  • What if, when I suggest a tool or a programming philosophy, it masks the need to dive deeply into other factors?

The more I researched my approach, I came across a concept that is vital to team effectiveness when solving complex problems: emotional intelligence.

Emotional Intelligence, sometimes referred to as EQ (emotional intelligence quotient) to complement IQ (intelligence quotient), is the ability to be aware of, express, control, reason, and interpret emotions appropriately.

Within a team, many, many, many studies have shown that EQ, more than IQ, is the key to solving complex problems.

The team dynamic is engrained with the DNA of open-source projects. Any Drupal issue queue or Packagist library commit log supports that.

The better question I ask myself, however, is:

  • Are the teams I work on the most emotionally intelligent?
  • If not, what am I doing to improve that metric?

Peeling back this onion revealed the societal constructs that affected how I view an effective team.

Typically, I look to the most technical people I know for answers. In some cases, I follow the stereotypical engineer playbook of positing a hypothesis, demanding evidence, and playfully browbeating a decision.

Put another way, how many times have I jokingly used the phrase “You are doing it wrong”? Is that the most effective solution, even when I mean no malice?

As the research suggested, this emotionally oblivious approach was philosophically incongruent with proven science!

  • How could I call myself an engineer?!?!
  • How could I obsess about the pursuit of efficiency and solution, when my own attitude was blunting my team’s effectiveness?

I needed to do better.

I needed to find something, rooted in math and science, that helped me understand how to refactor my way of thinking.

I then learned about perspective and heuristic techniques. Perspective is how one looks at a problem. Heuristic is the mental shortcut one uses to arrive at a solution. Both are shaped by experience and knowledge, but the nuance in process from a variety of individuals is key.

Dr. Scott Page elaborates :

Screen Shot 2014-09-03 at 11.36.31 AM

The diversity of an agent’s problem-solving approach, as embedded in her perspective-heuristic pair, relative to the other problem solvers is an important predictor of her value and may be more relevant than her ability to solve the problem on her own. Thus, even if we were to accept the claim that IQ tests, Scholastic Aptitude Test scores, and college grades predict individual problem-solving ability, they may not be as important in determining a person’s potential contribution as a problem solver as would be measures of how differently that person thinks.

It opened my eyes to how I’ve been going about solving complex problems all wrong.

In the context of a complicated problem, there is a higher likelihood of finding a global optimum (the best solution) when you have a diverse set of team members with local optimum (their best solution). Put simply, I needed to engage more (not less) with people who were different than me.

In essence, given the right conditions, diversity trumps ability!

What’s interesting about this research, however, is the fact that communication among members with different perspectives is very difficult.

In fact, as Dr. Page continues:

Screen Shot 2014-09-03 at 11.36.31 AM

Problem solvers with nearly identical perspectives but diverse heuristics should communicate with one another easily. But problem solvers with diverse perspectives may have trouble understanding solutions identified by other agents.

Thus, we’ve come full circle to why EQ is so important.

If team members are not in-tune with each other, the benefits gained from their diversity can be lost. It is vital, therefore, in my unabashed obsession to being an engineer, that I not only need to improve my EQ, but surround myself with colleagues who have a high EQ and learn from them.

So what are the characteristics of high EQ individuals? Statistically, who has high EQ?

Some of our thought leaders here at Phase2 have answered that question.

If you’re interested in learning more, find me as I share my ideas on a building a more inclusive community at various conferences and camps!

Profiling Drupal Performance with PHPStorm and Xdebug

Profiling is about measuring the performance of PHP code, at least when we are talking about Drupal and Xdebug. You might need to profile your site or app if you work at a firm where performance is highly scrutinized, or if you are having problems getting a migration to complete. Whatever the reason, if you have been tasked with analyzing the performance of your Drupal codebase, profiling is one great way of doing so. Note that Xdebug’s profiler does not track memory usage. If you want to know more about memory performance tracking you should check out Xdebug’s execution trace features.

Alright then lets get started! 

Whoa there cowboy! First you need to know that the act of profiling your code is itself taking resources to accomplish. The more work your code does, the more information that the profiler stores; file sizes for these logs can get very big very quickly. You have been warned. To get going with profiling Drupal in PHPStorm and Xdebug you need:

To setup your environment, edit your php.ini file and add the following lines:

Depending on what you are testing and how, you may want to adjust the settings for your site. For instance, if you are using Drush to run a migration, you can’t start the profiler on-demand, and that affects the profiler_trigger setting. For my dev site I used the php.ini config you see above and simply added a URL parameter “XDEBUG_PROFILE=on” to my site’s url; this starts Xdebug profiling from the browser.

To give you an idea of what is possible, lets profile the work required to view a simple Drupal node. To profile the node view I visited http://profiler.loc/node/48581?XDEBUG_PROFILE=on in my browser. I didn’t see any flashing lights or hear bells and whistles, but I should have a binary file that PHPStorm can inspect, located in the path I setup in my php.ini profiler_output_dir directive.

profiler_url

Finally lets look at all of our hard work! In PHPStorm navigate to Tools->Analyze Xdebug Profile Snapshot. Browse to your profiler output directory and you should see at least one cachgrind.out.%p file (%p refers to the process id the script used). Open the file with the largest process id appended to the end of the filename.

PHPStorm Cachgrind Picker

We are then greeted with a new tab showing the results of the profiler.

PHPStorm Xdebug Profiler

The output shows us the functions called, how many times they were called, and the amount of execution time each function took. Additionally, you can see the hierarchy of all function calls and follow potential bottlenecks down to their roots.

There you have it! Go wild and profile all the things! Just kidding, don’t do that.

How and Why to Patch a Drupal Module

Recently, at Drupal Camp Costa Rica, I was pleased to discuss what I feel is a very important, and very fundamental, Drupal technique. Patching code is something many of us have done, but the Drupal community has evolved standards around when, why, and how it’s appropriate to do so. I’d like to run through some of the highlights of that talk here.

What’s a patch?

Let’s start at the beginning. What do I mean by, “patch a module”? A patch is a small text file that contains a list of all the differences between the code as published (usually on drupal.org) and the code as you wish it to run on your site. You “apply” it to the original code, and the changes are made so that you can use the code the way you want it.

Why would I do that?

drupalkittenIn the Drupal community, we DON’T HACK CORE! This is a commonly held tenet of Drupal development, and there are very good reasons not to hack core, or even contrib modules:

  1. Forward compatibility: If new features or changes, or most importantly, security releases, are made to the module, you can’t take advantage of them without losing your changes.
  2. Developer friendliness: If your changes introduce a bug down the road, other developers will not look in that module, because they will assume it hasn’t been changed. This will cost them time and frustration.

What’s the difference?

What is the difference between a patch and a hack? Method.

When I say I “hack” a module, I mean that I am changing the module code directly, putting it straight into my site repo or on my site server, and running it. Changes like this are usually pretty invisible to other developers.

When I say I “patch” a module, it means that the changes that I’ve made are in a separate text file, which is applied to the module by when the site is built. These changes are also easily accessed and reviewed by other developers.

This tiny methodology difference means a great deal in actual practice.  A module that’s been hacked is very difficult to use in the long term. Changes made to it are often not recorded anywhere (or anywhere anyone would look), and if the module is replaced, by say a new or updated version, then those changes are lost forever.

It’s ok to patch core or contrib, just don’t hack it!

When would I patch a module?

  • You’ve found a module that does most of what you need… but not quite everything.
  • You’ve found a bug in the module.
  • You need to integrate custom functionality into the module, but it doesn’t have the right API functions.
  • You need a change right now and the module maintainer isn’t responding.

When would I not patch a module?

  • The module provides hooks or alter functions that will allow you to do what you need.
  • The module only does a little of what you need, and you probably can build a custom module for the same effort.
  • The dev version of the module has what you need, or there’s already a patch in the issue queue.

Please note this last point – work smarter! Spend some time before you get to coding to make sure that someone else hasn’t already done the work for you! That’s open source, folks – use it!

So, how do I do it?

Step 1: Check out the module

checkitoutThe easiest way to generate a patch file is by using git; this means you need to check out the module. See the screenshot here for instructions on checking out a Drupal module. Note, if you are running a development site, you can check this module out into your sites/all/modules directory; git is smart enough to handle it even if your full site directory is a git repo, too. (Though I don’t recommend that methodology, and you’ll see why later.)

Be sure to check out the dev version of the module; it’s going to be the most up-to-date, and module maintainers want to have your code apply to the latest version of what they’re working on.

Step 2: Hack away!

This is where you have the opportunity to make and test the changes you need to make in the code.

Step 3: Make the Patch

There’s two ways of doing this: on the command line, or via a GUI.

Command line is actually pretty easy. Once you’ve made your changes in the checked-out module directory, you run this simple command:

This will put all modified files into the patch. As with most command line tools, there are complicated options to only include certain files, or to compare different directories, etc. However, I personally favor a different way…

Sourcetree. It’s no secret that we here at Phase2 are huge Atlassian fans. This is another product by them, a GUI interface for management of your repositories, and it is awesome. It’s also free. It can manage your repos, help you keep them organized, and provides a visual interface for branching, git flow, and – of course – generating patches. Personally, while I can work with git on the command line, I don’t.  Because I use this instead.

Did I mention it’s free? (Disclaimer: I didn’t get anything for this recommendation, except a free copy of Sourcetree).

Congratulations! You’ve written a patch for a module! Now what?

Step 4: Submit your work

Issue QueueCreate a new issue in the issue queue of the module you’re working on. Fill out the form, but don’t attach your file just yet.  You’ll need to rename it, and the Drupal community has a specific formula for doing so.

[module name] – [short description] – [issue number] – [comment number] . patch

That’s the module name, followed by a one to two word description of what the patch does, followed by the nid of the issue queue node, and the number of the comment, dot patch.  Let me show you where to find those numbers.

find issue numberfind comment number The issue number is the nid of the issue node, and can be found in the url.

The comment number is not the cid of a comment; it’s the visual display number of the comment – see the second screenshot on the right. However, it’s the number of the comment you’re about to add, so when you rename your patch, rename it the number of the last comment, plus one.

So, let’s make an example.  For the module patch_demo, you need to add an additional field to the database table for “job roles”. The nid of the issue node is 2313551.  There are currently two comments on that thread. So, you could title your patch:

Now, make that comment and attach your patch!  Be sure to set the issue “Status” to “Needs Review”, as this will trigger drupal.org to automatically inspect and attempt to apply the patch.

Step 5: Bring it all together

Drush make is an amazing tool. It allows you to specify modules, themes, libraries, site structure… everything for building a site. While the use of it is pretty involved, and beyond the scope of this tutorial, I will touch on one thing – the ability to add a patch to your site build – automatically.

make file

This is a very basic drush .make file.  It tells drush to download drupal core, the features module, and the patch_demo module from my drupal.org sandbox.  The last line, outlined in orange, tells drush to grab that patch file from the drupal file server, and apply it to the patch_demo module. Note that part of the array specification includes the issue queue nid – this is important, as it allows future developers to easily find that issue queue and see why you are patching the module.  The naming convention on the patch file itself tells future devs which patch to look for, and in what comment.

Running drush make on this make file will download drupal, the features module, and my patch_demo module. It will assemble the site, then apply the patch to the patch_demo module, making the changes we specified much earlier in the process.

But whhhhhhhhy???

Since the patch only needs to be accessible to drush via the web, it could be anywhere – like, on a file server, or your company’s web site download directory, or Github, or Dropbox. Why contribute in the Drupal issue queue?

Because each of you, no matter how new, or inexperienced, or totally ninja, can help make Drupal better. Communities work best when everyone contributes. Your patch may not be accepted, or it may, or it may spark a discussion that leads to a better way of doing what you need. This is good for Drupal, which means in the long run it’s good for you too!

Also – even it it isn’t accepted, it’s in the issue queue file system, so it never goes away – meaning you can continue to use it. I’ve submitted many patches I knew would never get accepted, because I needed something to work in a particular unique way. That’s OK.

Finally, on a more personal note, contributing in the issue queues helps get you known in the Drupal community. This is great for your career.

Help make Drupal better. Save the kittens. Get yourself a better job. Contribute!

Linkage

Here are a number of really good resources I drew on to write this presentation and tutorial.

Last word: Thanks to the Drupal Camp Costa Rica team for giving me time to present such an important topic to their devs.  ¡Pura Vida!

Introducing OpenPublic’s App Strategy – Reimagining Content Management For Public Sector

Since it was first developed, OpenPublic has redefined open source public sector content management. Packaging government-focused functionality into a secure Drupal distribution, once a radical notion, is now an established open source web solution. As the foundation of many of Phase2’s public sector site builds,

OpenPublic has demonstrated the importance of solutions that are replicable, that can prevent duplication of services, and provide examples of repeatable best practices. OpenPublic serves as accelerator for building good government web sites and it contains best practices and features around mobility, security and accessibility.

With the release of OpenPublic candidate 1, we’re simplifying  content management in the enterprise public sector space by appifying all functionality and content types in OpenPublic. What was once a wilderness of modules and configuration will be encapsulated in a clean collection of Apps, making all OpenPublic’s out-of-the-box functionality simple to configure by non-technical site administrators. This new App strategy will make it easier and cheaper for governments to implement the web functionality they need.

So, what is an App?

It can be confusing to pin down definitions for terms like: modules, features, distributions, and Apps. An App is simply a collection of code (modules, features or themes) that delivers a distinct piece of functionality upon installation. Like the “apps” on your smartphone or tablet, Drupal Apps provide functionality for a specific purpose – quickly, easily, and independently.

In 2011, Jeff Walpole introduced the concept of Apps for Drupal distributions and the new Apps module. Apps improve usability for site administrators, particularly as compared to the traditional Drupal Configuration dashboard. From the beginning, Apps added extensibility and interoperability to Drupal. Now, instead of adding Apps for extensibility, we’re appifying all distribution functionality for OpenPublic 1.0, finally giving the content administrators full configuration control.

Appification of OpenPublic For State and Local Government

One Code Base, Many Departments

OpenPublic  Apps provide city, county, and state government agencies the ability to turn on and off independent pieces of functionality without affecting any other functionality on their platform. Many public sector agencies require a unified CMS spanning all departments and agencies. OpenPublic provides this through standard Apps developed for government needs, including directory management, stringent security, and editorial workflow. However, the flexibility of OpenPublic 1.0’s Apps also allows for specific functionality by department. This means that trying out new functionality is as easy as turning an App on or off, giving governments the opportunity to test innovative approaches without heavy risk or cost of implementation. See how San Mateo County uses OpenPublic Apps.

0c93dd3238b447ada9f6da67664bea85

Simplified Configuration

Apps take Drupal development out of the equation, empowering site administrators to skip technical development when configuring individual department sites. Each App is self contained, so changing the configuration does not cause problems with other site features.

 Custom Functionality Made Easy

With OpenPublic, users can develop Apps specific to their objectives. San Mateo County, Calif., for instance, used OpenPublic to develop an App which adds custom location mapping to any page on the web platform. Once created, the San mateo County team was able test their new App in one department, then enable it for other departments when it was deemed successful. The sky’s the limit with OpenPublic’s new App structure with unique and flexible functionality for public sector platforms.

OpenPublic  is breaking new ground in web experience for public sector site administrators and visitors alike. With the Appification of all functionality in OpenPublic, we are knocking down traditional barriers to Drupal site maintenance and scalability with intuitive configuration. Stop by Experience Director Shawn Mole’s Capital Camp session with Greg Wilson, Director of Public Sector Practice, to learn more about how OpenPublic  is truly the next generation of open source government sites.

 

Static Drupal – Taking the pain out of Drupal hosting

Think back with me to the days of static websites. Those were easy! You just threw some html up on a server and that was it. No performance issues, no security issues, no worries about redundant failover if something goes wrong. No getting woken up in the middle of the night because something did happen. It was simple, fast and hard to break.

All that changed with more Dynamic websites like Drupal. We get a lot for using Drupal. We get a first class CMS, huge ecosystem of modules and the ability to change content live on the server. But we also get a lot of headaches along with it. We have to design a server system that can handle the load, be constantly monitoring it for failures, apply security updates and be vigilant of security breaches and be ready to be woken up when something doesn’t work.

We recently were challenged to find a way to use the power of Drupal as a CMS but serve the site as generated static pages. It was an interesting idea and we decided to see if we could make it work. We started out by building a proof of concept module that would take a Drupal site and render it as plain HTML pages with all the associated assets such as JS, CSS and images. This turned out not to be too hard once we used a little apache rewrite rule to fix url paths. We decided to call this module the Static module. It is available on Drupal.org (http://www.drupal.org/project/static)

By itself this module turns out to be really useful for two use cases.

Static websites

There is a class of website that is built in Drupal but is rarely if ever updated. There are no “community” bits to it like comments and webforms. For this type of website, you can use the static module to periodically export your entire Drupal site to static html that can be uploaded to a simple server without PHP, MySQL, memcache or anything other than a web server. If changes need to be made, they can be made on a laptop or staging server and then exported and uploaded again. This allows both viewing the full site for acceptance testing before it goes live and all of the performance and security gains of a static website.

Archive websites

A similar type of website is an archival site such as the various DrupalCon and DrupalCamp websites. While they no longer need to be live Drupal sites or be updated, they should still be accessible to see the content that was there. In the same way as simple Static websites, archived websites can be generated, uploaded and then left alone.
Static websites are great and all but we can actually take it one step further. Turns out this module is also even better for building Hybrid websites.

Hybrid websites

A hybrid website is the primary reason we created the static module. We wanted a system where Drupal was installed behind a firewall and a static site was periodically generated and copied over to a public web server. This would allow content writers to create, update and delete content within Drupal which would track what is changing and within a short period of time it would be generated and transferred to the public facing site. Essentially it would have the power of content editing in Drupal but be hosted to the public as static HTML.
We got this system working and I’m going to share more about it at my talk during CapitalCamp/Drupal4Gov. Be sure to check it out if you are coming.