Make Your Product Vision Real – A Case for Incorporating Prototyping Into Your Next Project

As product designers and experience strategists, we research how people use systems and design products that tap into users’ natural behaviors. We want people to instinctively know how our product works.

Years of research into the human mind tells us that our brains love patterns, the repeated way in which something happens or is done. Our subconscious mind uses what we’ve learned from patterns – like turning a knob will open a door – to instinctively make decisions about what we do throughout our day. This is why we can walk or breathe without thinking about it – we spend most of our time running on autopilot.

We have an understanding of how people make decisions, but we forget to apply this knowledge when communicating our product vision to stakeholders.

There are Drawbacks to Designing in the Abstract

Experience design deliverables, or artifacts, are abstract. We too often produce artifacts, intended to build a shared understanding of a product vision, that are hard to understand. Low-fidelity wireframes and complex flow diagrams require stakeholders to think hard about what we are trying to communicate. They mentally fill in the gaps where we lack details. We consistently break Steve Krug’s number one rule: “Don’t make me think!”

Imagine how these abstract artifacts skew conversations about a product:

We show a stakeholder some wireframes and talk them through the features. Once they see them they begin to imagine the ways features will look and act based on similar products they have used.

While perfectly natural, this behavior is problematic – what we envision may be nothing like products this stakeholder has previously used. These assumptions your stakeholder makes will lead to you and your stakeholders having different expectations during product development.

You need to make artifacts as real as possible in order to elicit the most unbiased, unimpeachable feedback from users during research. You do not need to build a fully functioning product to validate your idea.You do need to eliminate or reduce the guesswork needed to understand how your product will work.

Make Your Product Vision Real

Prototyping is a great way to eliminate ambiguity so that you get the best results from user research. A prototype is a preliminary model of a product used to show a concept or validate an idea. A prototype should only contain the minimum amount of content, design and functionality needed to demonstrate how the end-product will function.

Context is key to determining fidelity of a prototype. If you are conducting user testing with a tech-savvy group of stakeholders, clickable wireframes may suffice. If you are introducing a new concept to a set of clients, then you may need a higher-fidelity, interactive web page. Your prototype should only contain the fidelity needed to have a meaningful conversation with your users about your product.

Build The Right Prototype For You

There are many different approaches to building prototypes. You can link wireframes together to show user flow with a system like inVision, or build interactive features using an open source CMS like Drupal.

When creating prototypes, make sure to include the following:

  1. The main actions that a user can take and the reactions they will receive from interactive elements.

  2. The key messages you want to communicate to users at different stages of their interaction.

  3. A programmatic way to track user behavior while they use the prototype.

Get Better Results from Your Projects

Some of the many benefits of prototyping are:

  • It produces more accurate results from user testing, allowing you to better determine what works and what doesn’t.

  • It gives you more opportunity to focus on interaction design by forcing you to have conversations about interactive elements during user research rather than development.

  • Prototypes bring less-apparent usability issues to light earlier in the development process.

  • You have a potential starting point to work from when beginning development, minimizing the amount of work that needs to be done in the long run.

John Whalen said “UX does not happen on a screen. It happens here. In the mind.” Keep that in mind (no pun intended) as you seek to build a shared understanding of, and validate, your product ideas. The more real you make the experience of interacting with your product early in the design process, the more accurate a feedback you will get from your users. For more thoughts on prototyping, check out Frederic Mitchell’s “Static Prototyping and Keeping Drupal Simple (KDS)” and “The Devil’s in The Details” by Sharon Smith!

Transitioning to Drupal 8 templates with Twig

As many of us know, Drupal 8 beta was released at the beginning of October which has given us a great preview of the goodies to come. One of those goodies is a new theme engine, Twig, which replaces PHPTemplate. Twig gives us a lot of extra power over our templates and allows us to clean up a lot of the extra cruft that PHPTemplate forced us to add.

So you may be asking, how will our brand new Drupal 8 templates look? Well they are going to be a lot leaner. The extra cruft of php tags, functions, and other miscellaneous bits are no longer needed – in fact, they don’t even work in Twig templates. Arguably, it will be easier to read at a glance what is happening in the Twig versions of Drupal templates. For a great example of how the new templates will look, we can crack open the Views module and compare the templates in the Views module 7.x version to the ones in the Views 8.x version. So let’s compare the views-view.tpl.php file from Views 7.x to views-view.html.twig file in Views 8.x.

Views-compare-part-oneViews-compare-part-two

Twig Comments

Starting from the top, lets work our way down to see what has changed. In the documentation section we can see that for the large block of comments, as well as single line comments, the {# #} syntax is used.

In the list of variables variables that are available to use, you may notice a few other changes. For example, Twig is not raw PHP, so the ‘$’ is not used at the beginning of variable names. Another change is that is that the $classes_array has been replaced by the attributes variable.

Rendering Variables

 Views-compare_example_one

Moving down to line #40 we can see the first instance of a variables rendered using Twig. The double curly bracket syntax, {{ my_variable }}, tells Twig that this is a variable and it needs to be rendered out. A variation of this syntax uses dot-notation to render variables contained in an array/object. This syntax looks like {{ my_variable.property_or_element }}, which makes it extremely easy to use. Dot notation is not used in this particular Twig template.

Twig Filters

Another powerful feature of Twig is template filters. Filters allow us to do simple manipulations of variables. The syntax for using a filter is similar to the syntax for rendering a variable. The difference is that after the variable name, you insert a | (pipe), followed by the name of the filter. For an example to make a string from a variable all lowercase you would do {{ variable_name|lower }} which transform all of the characters to lowercase. If you have used templating systems from other frameworks, such as Angular.js, this syntax may look familiar to you. This particular Views template does not use filters, but you can see examples of different filters on the Twig documentation site. If none of the predefined filters satisfy your requirements, you can extend Twig to create your own filter. The Twig documentation site provides details about creating your own custom filters..

Twig Logic

Views-compare_example_two

Jumping to line #42, we can see the {% %} (curly-bracket and percentage symbol) syntax, which is used for template logic, such as if statements. This syntax tells Twig that we are not rendering something, but rather that we need to process some logic, such as a control structure or loop, in our template.

The Takeaway

This blog post is a high level overview of how Twig templates in Drupal 8 will look.  With Twig, we can choose to use the out-of-the box tools it provides, or we can dive in and extend it with additional features such as new filters. For more information I would highly recommend reading through the Twig documentation for designers and for developers.

BADCamp Sprinting Success Story : Drush make files support YAML

After a very successful drush code-sprint at BADCamp 2014, drush make now supports YAML format!

Instead of the old INI format

YAML can be used with the latest version of Drush 7:

Included .make files whether local, or discovered recursively within downloaded projects, can be in either YAML of INI format.

In order to use the newly-supported YAML format, simply name files with a .yml extension, such as my_project.make.yml.

The best part? This can be used now! Even though YAML files are mostly a new concept for Drupal 8, drush make will parse YAML make files for Drupal 7, and even Drupal 6. Want to learn more about DRUSH make files? Check out Joe Turgeon’s “Getting Started With Grunt Drupal Tasks

AngularJS Meet Open Atrium

atrium-logo (1) The recent 2.23 version of Open Atrium contains a cool new interactive site builder and navigator application (see it in action). This application was written using the AngularJS framework.  The combination of Drupal, jQuery, and AngularJS proved to be powerful, but wasn’t without some pitfalls.

Using AngularJS in Drupal

AngularJS-large The basics of using Angular within Drupal is pretty straight-forward.  Simply reference the external AngularJS scripts using the drupal_add_js() function, then add your custom javascript app code, then use a tpl template to generate the markup including the normal Angular tags.  For example, here is the Drupal module code, javascript and template for a simple Angular app:

Now, obviously we aren’t using the full Angular framework here.  We aren’t using any directives, nor are we really using Angular as a MVC framework.  But it gives you the idea of how easy it is to get started playing with basic Angular functionality.

Angular plus jQuery

Developing javascript applications in Angular requires a different mindset from normal Drupal and jQuery development.  In jQuery you are often manipulating the DOM directly, whereas Angular is a full framework that allows data to be bound and manipulated on page elements.  Trying to combine both is often a source of frustration unless you understand more about how Angular works behind the scenes. Specifically, Angular has it’s own execution loop causing a mix of Angular and jQuery code to not seem to execute in a straightforward order.  For example, in the above code, we set the class of the H3 based on the current “count” variable.  What if we modified the updateCount function to try and set a css property for this class:

If you click the button you’ll notice that the css color does NOT change to red! The problem is that Angular is executing the query function call BEFORE it actually updates the page.  You need to delay the jQuery so it executes after the current Angular event loop is finished.  If you change the code to:

then it will work.  The timeout value can be anything greater than zero.  It just needs to be something to take the jQuery execution outside the Angular loop. Now, that was a horrid example!  You would never actually manipulate the css and class properties like this in a real application.  But it was a simple way to demonstrate some of the possible pitfalls waiting to trap you when mixing jQuery with Angular.

Drupal Behaviors

When doing javascript the “Drupal way”, you typically create a behavior “attach” handler.  Drupal executes all of the behaviors when the page is updated, passing the context of what part of the page has changed.  For example, in an Ajax update, the DOM that was updated by Ajax is passed as the context to all attached behavior functions. Angular doesn’t know anything about these behaviors.  When Angular updates something on the page, the behaviors are never called.  If you need something updated from a Drupal behavior, you need to call Drupal.attachBehaviors() directly.

Angular with CTools modals

In the Open Atrium site map, we have buttons for adding a New Space or New Section.  These are links to the Open Atrium Wizard module which wraps the normal Drupal node/add form into a CTools modal popup and groups the fields into “steps” that can be shown within vertical tabs.  This is used to provide a simpler content creation wizard for new users who don’t need to see the full node/all form, and yet still allows all modules that hook into this form via form_alters to work as expected. The tricky part of this is that as you navigate through the sitemap, Angular is updating the URLs of these “New” links.  But CTools creates a Drupal Ajax object for each link with the “ctools-use-modal” class in it’s Drupal behavior javascript.  This causes the URL of the first link to be cached.  When Angular updates the page and changes the link URLs, this Ajax object cache is not updated. To solve this within the Open Atrium Sitemap app, an event is called when the page is updated, and we update the cached Ajax object directly via the Drupal.ajax array. This was a rather kludgy way to handle it.  Ultimately it would be better to create a true Angular “Directive” that encapsulates the CTools modal requirements in a way that is more reusable.

Summary

Angular can be a very useful framework for building highly interactive front-ends.  Using Drupal as the backend is relatively straight-forward.  Angular allowed us to create a very cool and intuitive interface for navigating and creating content quickly within Open Atrium far easier than it would have been in jQuery alone.  In fact, we began the interactive site map tool in jQuery and the code quickly became unmanageable.  Adding functionality such as drag/drop for rearranging your spaces and sections would have been a mess in jQuery.  In Angular it was very straight-forward. Once you understand how Angular works, you’ll be able to blend the best of Drupal + jQuery + Angular into very rich interfaces.  Programming in Angular is very different.  Learn more about Angular on the Phase2 blog!

Hallway to the Future: Reflections of a Drupal Enthusiast

Before we go to the future, a little detour to the past…

Enjoy the view from the top of the center of knowledge and mysticism of the Incas, facilitated by the incredible #DrupalPicchu.

Seven years ago, I decided to go in a new direction. I left the non-profit I’d been working for, which focused on my exact academic interests, to join an entertainment company. There, I would be a part of  a team building and using an internet system covered in nodes, blocks, exposed integers and pagers that run on computer time: Drupal.

As I looked into the future, this modular open source “content holder software” seemed poised to be a springboard for a lot of smart, kind, passionate people to come together and build something greater than themselves.

Community is Key

Fueled by meetups, the community has a very involved distributed culture and communication network in which people form alliances around ideas.  There were a lot of aspects reminiscent of some of the indigenous communities funded by the NGO where I’d previously worked. The center of both was “the community” or in Spanish “la communidad.”  Even in a big company, the community is a major part of the story when open source is involved.

Screen Shot 2014-09-04 at 11.09.45 AM

Some Sony connected Drupalers at DC Chicago. Photo credit: Thomas Turnbull

About 2.5 years ago, I started a new chapter at Phase2, continuing along the path that I had started when I decided to see what all the Drupal fuss was about.

Screen Shot 2014-09-04 at 11.10.46 AM

Phase2 orange related sign I saw in a cornfield.

As one of the leaders in open source and big Drupal, Phase2 has a high concentration of smart, talented and passionate people in addition to the amazing clients we work with to build amazing open source web systems.

DSCF1631

Phase2 team at DC Austin

It’s All Happening.

Seven years later, we stand on the cusp of a very interesting time for open source. Drupal powers a key segment of the web, and its societal power to influence culture is outstanding. You can petition the president, donate to poverty fighting in NYC, buy a bicycle, keep up to date with latest science news or local news,  watch college sports, discuss servers and what software lives on them, coordinate the large amount of information on a humanitarian crisis, catch the latest music video, decide what to major in at your University, visualize a social movement, hear about the latest in international crowd funding,  and launch your own indigenous digital asset management library.

Drupal is not only the enterprise system of record for open source, but it is growing in leaps and bounds throughout the world, supported by its sophisticated multi-lingual underpinning. Over the last year, I’ve had the incredible opportunity to get to know the international community even more through attending and speaking at several amazing camps in South and Central America: Drupal Summit Loja, Drupal Camp Costa Rica, Drupal Picchu & Drupal Camp Mexico City.

Screen Shot 2014-09-04 at 11.14.45 AM

Drupal Picchu group shot via Cristian Torres

Right now, Drupal powers academic institutions, Fortune 500 platforms, and global non-profits. This is Drupal. This is what we have to start from! Not to mention Drupal’s greatest asset: the incredible community of people that develop its core, use it to get their content to the world, make it easy to host, consult on strategies, and share approaches with others.

Looking to the Future of Open Source…

Just imagine – if all of the above has happened in the last seven years, what will the next 11 bring?

While the technical advances in Drupal 8 are amazing, it’s the implications for the communities around Drupal which I feel most connected to.

  • We are becoming more accessible to non-English languages through the #D8MI (Drupal Multilingual headed up by Gábor Hojtsy). This will allow Drupal to become more globalized than it already is, opening up our ability to interact with even more communities around the world.

  • We are becoming more technically accessible as well accessible. By Drupal adopting HTML5, we make it easier to create responsive, and accessible websites for a greater range of abilities and technical capacities.

  • Through the ideas in the Drupal as a RESTful data store it will become even easier for us to integrate with other software projects, making us more collaborative, and creating a more positive and open world.

As Drupal continues to evolve in the open source space and expand globally, I hope to see more inspiring stories like this one of a community using open source technology to build and operate their own cellular networks in rural Mexico.

I look forward to continuing to engage and be a part of this awesome community, and I’d like to hear your story as well. How did you get involved in our eclectic and awesome community? In what ways are you most excited to see Drupal evolve in the future?

If you are in the Bay Area this week, check out the Bay Area Drupal Camp (BADCamp) November 6th-9th.  This completely free, volunteer run 4 day conference is yet another example of the power of open source community.  Join me at the BADCamp non profit summit tomorrow to learn how nonprofits are leveraging Drupal to further their organization’s missions and ultimately contribute to our shared global community.

And for those international adventurers and folks in Latin – the first DrupalCon Latin America is February in Bogota, Colombia #vamosalfuturo.

 

Getting Started with Grunt Drupal Tasks

In September, Phase2 released a Grunt-based tool for building and testing Drupal sites. We have been working on the tool since January, and after adopting it as part of our standard approach for new Drupal sites, we wanted to contribute it back to the community. We are happy invite you to get started with Grunt Drupal Tasks.

Grunt is a popular JavaScript-based task runner, meaning it’s a framework for automating tasks. It’s gained traction for automating common development tasks, like compiling CSS from Sass, minifying JavaScript, generating sprites, checking code standards, and more. There are thousands of plugins available that can be implemented out-of-the-box to do these common tasks or integrate with other supporting tools. (I mentioned that this is all free and open source software, right?)

Grunt Drupal Tasks is a Grunt plugin that defines processes that we have identified as best practices for building and testing Drupal sites.

Building Drupal

The cornerstone of Grunt Drupal Tasks is the “build” process, which assembles a runnable Drupal site docroot from a Drush make file and custom code and configuration.

The make file defines the version of Drupal core to use, the contrib modules, themes, and libraries to download, and even patches to apply to any of these components. The make file can include components released on Drupal.org or stored in public or private repositories. For patches, our best practice is to reference patches hosted on Drupal.org and associated with an issue. With these options, the entire set of components for a Drupal site can be declared in a make file and consistently retrieved using Drush.

After the Drush make process assembles all external dependencies for the project, the Grunt Drupal Tasks build process adds custom code and configuration. This includes custom installation profiles, modules, and themes, as well as “sites” directory files, like sites.php and settings.php for one or many subsites, and other “static” files to override, like .htaccess and robots.txt. These custom components are added to the built docroot by symlink, so it is not necessary to rebuild for every update to custom source code.

These steps results in a Drupal docroot assembled from custom source in the following structure:

Grunt Drupal Tasks includes other optional build steps, which can be enabled as needed for projects. One such task is the “compile theme” step will compile Sass files into CSS.

This build process gives us a reliable way for assembling Drupal core and contrib components, for adding our custom code, and integrating development tools like Sass. By using Grunt to automate this procedure, it becomes a portable script that can be shared among the project’s developers and used in deployment environments.

Testing Drupal

In order to help make best practices the default, Grunt Drupal Tasks includes support for a number of code quality and testing tools.

A “validate” task is provided that includes checking basic PHP syntax and Drupal coding standards using PHPLint and PHP Code Sniffer. We highly recommended that developers use this command while coding, and have included it as part of the default build process.

An “analyze” task is also provided, which adds support for the PHP Mess Detector. This task may be longer-running, so it is better suited to run as part of a continuous integration system, like Jenkins.

Finally, a “behat” task is provided for running test scenarios with Behat and the Drupal Extension. This encourages writing Behat tests for the project and committing them with the project code and build tools, so the tests can be run by other developers and in the integration environment by a continuous integration system.

Scaffolding for Drupal Projects

The old starting point for Drupal projects was a vanilla copy of Drupal core. Grunt Drupal Tasks offers scaffolding for Drupal projects that starts with Drush make, integrates custom code and overrides, and provides consistent support for a variety of developer tools.

This scaffolding is provided through the example included with Grunt Drupal Tasks, which is the recommended starting point for new projects. The scaffold structure adds a layer above the aforementioned “src” directory; this layer includes code and configuration related to Grunt Drupal Tasks (Gruntconfig.json and Gruntfile.js), dependencies for the supporting tools (composer.json), and other resources for the tools (features/, behat.yml, and phpmd.xml).

The example includes the following:

For full documentation on starting a new project with Grunt Drupal Tasks, see CONFIG.md.

Learning More

Watch the Phase2 blog for more information about Grunt Drupal Tasks. If you are attending the Bay Area Drupal Camp this week, please check out my session on Using Grunt to Manage Drupal Build and Testing Tools.

Introducing the Behat Drupal Extension 3.0

I am proud to announce the release of the Behat Drupal Extension 3.0! Since 2012, the Drupal Extension has been used for Behaviour Driven Development of thousands of Drupal sites all over the world. The project began as part of the Drupal.org upgrade, and was quickly generalized to enable the testing of any Drupal site.

Version 3 has many exciting new features, and is compatible with Behat 3. Note version 2 does not exist, wanting to avoid version soup.

Behavior Driven Development at BADCamp

Before I dive into the new features, it is important to clarify the difference between testing an application, and Behavior Driven Development (BDD).

As everzet pointed out at Drupalcon no business-critical feature starts with

rather, it starts with a conversation

While both are testing something, only the latter is truly describing a behavior that matters to a stakeholder. This is the subject of a blog post, or series of posts, in and of itself, so more on this subject later.

What’s new

Documentation

Documentation for this project has been integrated into the repository, and is automatically building on readthedocs.org. Thanks to Melissa Anderson for this awesome work!

A starter context

A starter Drupal context that makes no assumptions around the language used for each test, but still provides all the previous functionality to interact directly with Drupal.

Users are encouraged to start tests from this context which will allow them to use truly ubiquitous language that is specific to each project.

Drupal Drivers

The Drupal Drivers now exist in a separate project, allowing for non-Behat applications to interact with Drupal (e.g., calling directly from Mink, or Codeception).

Note that the Drupal 6 driver has been removed, but since drivers are now separate projects, it will be easy to port that over to the Drupal Extension 3, should somebody want.

It should also be noted that Drupal 8 support will require ongoing work as the code base there evolves towards release.

More granular pre-defined step-definitions

Existing step definitions have been split into 4 indepentent contexts:

  • DrupalContext – This contains steps for working with content, users, and taxonomies.
  • MinkContext – This is an extension to the Mink Extension, providing additional steps for working with regions and forms.
  • MessageContext – Provides steps for working with Drupal success/warning/error messages.
  • DrushContext – Provides steps for calling drush commands directly from scenarios

This allows for the use of some pre-definied step-definitions, rather than the previous all-or-none approach.

No more regex!

The pre-definied steps now use the new turnip syntax introduced in Behat 3:

rather than

What’s a ‘node’?!

The term node has been removed from steps and replaced with content in all pre-defined steps.

What’s next?

BDD Mini Summit

I will be attending the Behat mini-summit at BADCamp along with several other folks from Phase 2. I hope to highlight the Drupal Extension 3, and discuss best-practices for the wide variety of testing needs. I also hope to continue discussions around Behat, Mink and Drupal core.

Simplify Your Logstash Configuration

As I mentioned in my recent post, I got a chance to upgrade the drupal.org ELK stack last week. In doing so, I got to take a look at a Logstash configuration that I created over a year ago, and in the course of doing so, clean up some less-than-optimal configurations based on a year worth of experience and simplify the configuration file a great deal.

The Drupal.org Logging Setup

Drupal.org is served by a large (and growing) number of servers. They all ship their logs to a central logging server for archival, and around a month’s worth are kept in the ELK stack for analysis.

Logs for Varnish, Apache, and syslog are forwarded to a centralized log server for analysis by Logstash. Drupal messages are output to syslog using Drupal core’s syslog module so that logging does not add writes to Drupal.org’s busy database servers. (@TODO: Check if these paths can be published.) Apache logs end up in/var/log/apache_logs/$MACHINE/$VHOST/transfer/$DATE.log, Varnish logs end up in/var/log/varnish_logs/$MACHINE/varnishncsa-$DATE.log and syslog logs end up in /var/log/HOSTS/$MACHINE/$DATE.log. All types of logs get gzipped 1 day after they are closed to save disk space.

Pulling Contextual Smarts From Logs

The Varnish and Apache logs do not contain any content in the logfiles to identify which machine they are from, but the file input sets a path field that can be matched with grok to pull out the machine name from the path and put it into the logsource field, which Grok’s SYSLOGLINE pattern will set when analyzing syslog logs.

Filtering on the logsource field can be quite helpful in the Kibana web UI if a single machine is suspected of behaving weirdly.

Using Grok Overwrite

Consider this snippet from the original version of the Varnish configuration. As I mentioned in my presentation, Varnish logs are nice in that they inclue the HTTP Host header so that you can see exactly which hostname or IP was requested. This makes sense for a daemon like Varnish which does not necessarily have a native concept of virtual hosts (vhosts,) whereas nginx and Apache default to logging by vhost.

Each Logstash configuration snippet shown below assumes that Apache and Varnish logs have already been processed using theCOMBINEDAPACHELOG grok pattern, like so.

The following snippet was used to normalize Varnish’s request headers to not include https?:// and the Host header so that therequest field in Apache and Varnish logs will be exactly the same and any filtering of web logs can be performed with the vhost andlogsource fields.

As written, this snippet copies the request field into a new field called full_request and then unsets the original request field and then uses a grok filter to parse both the vhost and request fields out of that synthesized full_request field. Finally, it deletesfull_request.

The original approach works, but it takes a number of step and mutations to work. The grok filter has a parameter calledoverwrite that allows this configuration stanza to be considerably simlified. The overwrite paramter accepts an array of values thatgrok should overwrite if it finds matches. By using overwrite, I was able to remove all of the mutate filters from my configuration, and the enture thing now looks like the following.

Much simpler, isn’t it? 2 grok filters and 3 mutate filters have been combined into a single grok filter with two matching patterns and a single field that it can overwrite. Also note that this version of the configuration passes a hash into the grok filter. Every example I’ve seen just passes an array to grok, but the documentation for the grok filter states that it takes a hash, and this works fine.

Ensuring Field Types

Recent versions of Kibana have also gotten the useful ability to do statistics calculations on the current working dataset. So for example, you can have Kibana display the mean number of bytes sent or the standard deviation of backend response times (if you are capturing them – see my DrupalCon Amsterdam slides for more information on how to do this and how to normalize it between Apache, nginx, and Varnish.) Then, if you filter down to all requests for a single vhost or a set of paths, the statistics will update.

Kibana will only show this option for numerical fields, however, and by default any data that has been parsed with a grok filter will be a string. Converting string fields to other types is a much better use of the mutate filter. Here is an example of converting the bytes and the response code to integers using a mutate filer.

Lessons Learned

Logstash is a very powerful tool, and small things like the grok overwrite parameter and the mutate convert parameter can help make your log processing configuration simpler and result in more usefulness out of your ELK cluster. Check out Chris Johnson’s post about adding MySQL Slow Query Logs to Logstash!

If you have any other useful Logstash tips and tricks, leave them in the comments!

 

DrupalCon Amsterdam Roundup

As I write this, I’m on a plane back to the US after a whirlwind 10 days in Amsterdam for DrupalCon Amsterdam 2014. As always it is so gratifying to meet and work with the international Drupal community. I love getting to take a look at what everyone is working on and collaborate with people that you might otherwise only know as IRC nicks. Here are my highlights of DrupalCon Amsterdam:

Sprints and Drupal.org Logging

I volunteer my time to help the drupal.org infrastructure team with their logging infrastructure. I was very happy to be able to sprint for 3 days, mainly on drupal.org infrastructure. On Sunday, Friday, and Saturday, I worked with the Drupal.org Puppet configuration to get a new CentOS 6-based log aggregation host ready to go running the latest versions of the ELK (ElasticSearch, Logstash, and Kibana) stack. The new Logstash configuration that we’ll be rolling out is much simpler. Stay tuned to the blog for information on some of the improvements we made in the process. We hope to deploy the new logging host within the week.

DevOps Meetup

Tuesday evening, I attended the DevOps Amsterdam meetup.  The meetup was sponsored by ElasticSearch, who bought a delicious dinner for all attendees as well as some drinks. During dinner, I sat with some folks from Germany and had a chance to speak with a number of ops-attuned folks about Docker and the possible use cases for it.

The meetup had some great content. There was a talk on how GitLab uses omnibus to package GitLab with far less hassle, a talk from fellow d.o infra team volunteer Ricardo Amaro on building the next-gen Drupal testbot on Docker, and a great talk from the DrupalCon Amsterdam DevOps chair Bastian Widmer on developing culture and sharing knowledge in an agency.

LSD Leadership Meeting

Earlier this year, Phase2 contributed some of its innovation hours to the LSD project and worked with Acquia to present a webinar on Behat and release a pre-built virtual machine designed to make it easy to start doing automated testing using Behat. During the LSD leadership meeting I joined Melissa Anderson of Tag1 Consulting and Hugh Fidgens of Pfizer in a breakout session discussing Behat. Quite a few organizations present were very interested in how they could use Behat to enable a behavior-driven development workflow with their developers, or to develop a good set of automated tests that could be run either as smoke tests or as end-to-end integration tests.

Behat Everywhere

In addition to the LSD leadership meeting, automated testing and BDD were topics on everyone’s minds throughout DrupalCon.

I attended 2 BoF about automated testing or Behat, the Testing Drupal 8 BoF, and Hugh Fidgen’s Organizational Behat BoF. These talked about how we could better leverage automated testing tools in Drupal core and in client work we build today, respectively. Many people have had some success in getting automated tests or a BDD workflow started, but there was a lot of talk about writing good sustainable tests and how to integrate these tools into your workflow.

Speaking of writing sustainable tests, my favorite session of the conference was definitely the session by Konstantin Kudryashov (the creator of Behat and Mink) on how to do BDD with Behat. The session was remarkable and left an impact on many folks who went there. It really emphasized the point that BDD must be about identifying and delivering business value in our projects, and that doing that is the way to write good tests. It is definitely worth an hour of your time to watch.

As testing best practices are refined in the Drupal community as well as in Phase2, I’m very excited that the talented Jonathan Hedstrom has joined the Phase2 team.  Jonathan is a maintainer of the Behat DrupalExtension, and is sure to help us further refine our testing practices as Phase2. Jonathan has been doing work recently on upgrading the DrupalExtension to support Behat 3 and has plans to generalize the drivers that the Behat DrupalExtensions provides so that it could be used with Mink for writing tests in straight PHP without needing to use Behat.

My “Open Source Logging and Monitoring Tools” Session

On Wednesday I presented my session “Using Open Source Logging and Monitoring Tools.” I covered a lot of information in my session including using Logstash, ElasticSearch, and Kibana.  Thanks to the tireless work of the DrupalCon A/C crew, the video recording of my session is online, and the slides are available on SlideShare if you would like to follow along.  The session had an excellent turnout, and there were some great questions and discussions following my session. I was quite pleased at the large turnout, and so was @KrisBuytaert, who been bringing information about DevOps-related topics to DrupalCons for years.

 

 

We were also fortunate to have Leslie Hawthorne of ElasticSearch in the audience. She gave out ElasticSearch ELKs to sprinters at the Friday event.

 

Takeaways

Based on the sheer number of people interested and sessions to devoted to both topics, it is clear that there is a growing interest in both logging and metrics and automated testing or BDD in the Drupal Community. This is also the 11th DrupalCon I’ve been to, and this year’s keynotes were the best I could remember. I really enjoyed Dries’s ideation around sustainable methods for getting contributions to Drupal core and contrib, and getting to see Cory Doctorow speak live about the perils of DRM and restrictions software freedoms was also excellent.

This is an exciting time for Drupal. Drupal 8 beta 1 is live. The community is active and engaged around making Drupal better, both by contributing to Drupal 8 and by doing a better job testing projects built on it. The DA and an army of volunteers have made huge strides on improving the infrastructure around core testing as well as around all the online communities under drupal.org.

Introducing DrupalCon Amsterdam Hackathon Champs!

Last Monday, Phase2 hosted a distribution hackathon at DrupalCon Amsterdam. While we were excited to see what our hackers would come up with, we were blown away by what was accomplished in an evening of hacking.  The winners of the hackathon developed a Drupal 8 distribution called Drupal Promo Kit. This distribution allows anyone to easily create presentations and landing pages using Drupal. Other hack projects included Open Atrium Apps, Panopoly apps, and more. I got to chat with the hackathon winners: Kate Marshalkina , Konstantin Komelin, John Ennew, and Mariano Barcia, to learn how they came up with their idea and how they did it:

Hackathon Winners

Q: What was your inspiration for your hackathon project?

A: We wanted to test our skills in Drupal 8 and see if we could build our own distribution.  We also wanted to build a unique solution that was useful and meaningful.

Q: How did you prep for the hackathon?

A: About a week before the hackathon Kate and I (Konstantin) brainstormed potential hackathon ideas. Our goal was to decide on a project that we could complete by the end of the hackathon, so once we decided on our idea for the project, we decided on the scope of our project and the minimal and maximum viable product.  We then decided on the tools we would use including Bitbucket, Google docs, Slack, and Trello.

Q: And once you arrived at the hackathon, what was your experience working with other people?

A: When the hackathon kicked off, we announced our project and goals to everyone and found some new team members that were interested in what we wanted to do. John Ennew and Mariano Barcia joined us and it was amazing how easy it was to pick up and start developing with other Drupalers. We were able to start developing rapidly in just a couple of hours. It didn’t matter that we were all coming from different backgrounds, with different skill levels, we were all speaking Drupal, a universal language.

Q: What do you think is the value of hackathons?

A: Hackathons are a great opportunity to meet other Drupalers in the greater Drupal community and work together to solve problems. Hackathons at Drupalcon are special because your  team can come from all over the world, and you all have different cultures, different jokes, and new ideas.  Not only do you build friendships throughout the event, but all the different perspectives and experiences strengthen your project. Hackathons let you dive into something you are interested in and at the same time, it can help push Drupal forward.  We learned so much about Drupal 8 while working on this project, and we plan on taking our experience and feedback to the core team to help improve Drupal 8.

—————————-

Interested in learning more about Drupal Promo Kit? Check it out on Drupal.org! We want to thank all the hackers that participated in our hackathon, and can’t wait to hack with our brilliant community again soon!  See more images of the hackathon and other DrupalCon Amsterdam photos on our Flickr!

By4dnAPCAAA-qSo