Sep 03 2015

Is this what humanity has become?: A brief look at the Syrian Refugees


If you’ve been watching the news over the past days and weeks, you’ve certainly seen information regarding the thousands of people fleeing Syria (and other nations in the Middle East [such as Afghanistan] and northern Africa [such as Eritrea]). Natives of these countries have been trying to escape due to the vast political and ethical injustices amidst their war-torn homes.

I had read all the numbers regarding the immigrants, and how the sheer amount of people trying to get into the EU (and possibly seek asylum) was crippling transportation systems, and could be damaging to the economy. A quick recap shows that approximately 350,000 people have migrated into the EU since the start of 2015, and that nearly 100,000 of them hailed from Syria (for a full look at the numbers, and an overview of the situation, please see the “Why is EU struggling with migrants and asylum?” article at BBC News). Looking at the numbers and reading about the conflicts and struggles, though, does not really illuminate the tragedy associated with the events. Though I generally don’t like the hype and drama that several media outlets try to foster, I think that in situations like this one, seeing evidence of the pain and suffering can bring the abominable nature to light.

One such story has been about a Syrian toddler named Aylan Kurdi. A photo of him and his older brother, Galip, can be seen below.

3-year-old Aylan Kurdi with his brother Galip

He and his family were trying to flee their hometown of Kobani, Syria and seek asylum in the EU. They, like many others, were trying to get away from the conflict, the war, and the brutality running rampant in their home. Unfortunately for little Aylan, he would never get to see this new life in the EU. Instead, he drowned, and his body was found washed up on the shore near the town of Bodrum, Turkey.

Body of drowned Syrian toddler Aylan Kurdi

He was trying to survive when the boat that his family was on capsized on its way to the Greek island of Kos. It goes without say that his family (including his father, Abdullah, who survived the sinking) is trying to endure the unimaginable despair that comes with the loss of a child. The pain extends much further, though. For instance, the downcast and devastated look on the face of the Turkish paramilitary officer who found the body speaks volumes.

Turkish officer carrying the body of Syrian toddler Aylan Kurdi

What was the reason for the loss of this little one? To briefly cover it—which unfortunately detracts from the severity of it—the people of Syria started anti-government protests back in 2011. The protests turned violent as supporters of President Bashar al-Assad and those who opposed his rule started fearing for their safety. The full-on civil war became drastically worse in August of 2013 when chemical weapons were fired into suburban and rural areas surrounding the capital city of Damascus. The principal causes of the conflict were political and religious in nature (fighting for democracy against a tyrannical ruler, and going against the Islamic State).

So, due to strongly-held political and religious beliefs, thousands of people have lost their lives whilst myriad others have had their dignity stripped from them. Even with involvement from outside nations, the conflict in Syria has not been adequately resolved, and may not be for many years to come. British Member of Parliament, Nadhim Zahawi posted on his Twitter feed a personal apology to Aylan Kurdi:

Twitter post by British MP Nadhim Zahawi about failing Syria

More important than the particular apology were the words that started his tweet, and the acronym with which he closed it. He opens with “We r [sic] nothing without compassion.” When we get so tangled up in our beliefs (political, religious, or otherwise) that we forget to love one another, we have lost sight of our own humanity. He then closes with “RIP,” which, at first glance, doesn’t sound of out the ordinary. However, it struck me because it stands for “rest in peace.” At a mere three years of age, why wasn’t little Aylan allowed to “LIVE in peace?” Matter of fact, why is anyone disallowed the opportunity to “live in peace” instead of “resting in peace” after death?

In my opinion, the particular causes for the rift in Syria are not where we should be focusing our attention. Instead, we need to remember what the true cost is: the loss of human life, livelihood, and personal freedoms. Again in my opinion, we as people should not hold beliefs of any kind so strongly that we forget about the dignity of others. Protesting to uphold our beliefs, struggling to preserve our ways of life, and pushing to uphold personal freedoms are all noble causes, but they should never supplant our fundamental responsibility of treating other people with respect and love. That is my main point. So, I applaud the media outlets for making the Syrian conflict “real” by showing photos that truly represent the devastating effects of war, instead of just spewing numbers that abstract us from the reality of it. However, they still seemed to miss the key point that nothing warrants the loss of human life.

I realise that this post is difficult to read and view, especially because of the horrific images, but the improper treatment of people is a topic about which I feel very strongly. I would like to end the post with something a little more positive regarding the migrants, though, and show an image that indicates it isn’t all terrible. The photo below is of a Bavarian officer who managed to get a young refugee boy to smile as he entered the capital city of Munich.

Bavarian soldier makes a young Syrian refugee boy smile as he enters Munich
Click to enlarge

Please remember that, no matter how passionately you hold your beliefs, nothing is worth hurting another person. Buddha taught that there is an unalterable law that “Hatred does not cease through hatred at any time. Hatred ceases through love.” Love one another, and only then will these types of inhumanities stop.

–Zach

 

Hatred does not cease through hatred at any time. Hatred ceases through love. This is an unalterable law. –Buddha

Aug 06 2015

Apache 2.4, the Event MPM, PHP via mod_proxy_fcgi and PHP-FPM – with vhosts!

Recently, I’ve spent a huge amount of time working on Apache and PHP-FPM in order to allow for a threaded Apache MPM whilst still using PHP and virtual hosts (vhosts). As this article is going to be rather lengthy, I’m going to split it up into sections below. It’s my hope that after reading the article, you’ll be able to take advantage of Apache’s newer Event MPM and mod_proxy_fcgi in order to get the best performance out of your web server.

1. Definitions:

Before delving into the overarching problem, it might be best to have some definitions in place. If you’re new to the whole idea of web servers, programming languages used for web sites, and such, these definitions may help you become more acquainted with the problem that this article addresses. If you’re familiar with things like Apache, PHP, threading, preforking, and so on, feel free to skip this basic section.

  • Web server – on a physical server, the web server is an application that actually hands out (or serves) web pages to clients as they request them from their browser. Apache and nginx are two popular web servers in the Linux / UNIX world.
  • Apache MPM – a multi-processing module (or MPM) is the pluggable mechanism by which Apache binds to network ports on a server, processes incoming requests for web sites, and creates children to handle each request.
    • Prefork – the older Apache MPM that isn’t threaded, and deals with each request by creating a completely separate Apache process. It is needed for programming languages and libraries that don’t deal well with threading (like some PHP modules).
    • Event – a newer Apache MPM that is threaded, which allows Apache to handle more requests at a time by passing off some of the work to threads, so that the master process can work on other things.
  • Virtual host (vhost) – a method of deploying multiple websites on the same physical server via the same web server application. Basically, you could have site1.com and site2.com both on the same server, and even using the same IP address.
  • PHP – PHP is arguably one of the most common programming languages used for website creation. Many web content management systems (like WordPress, Joomla!, and MediaWiki) are written in PHP.
  • mod_php – An Apache module for processing PHP. With this module, the PHP interpreter is essentially embedded within the Apache application. It is typically used with the Prefork MPM.
  • mod_proxy_fcgi – An Apache module that allows for passing off operations to a FastCGI processor (PHP can be interpreted this way via PHP-FPM).
  • PHP-FPM – Stands for PHP FastCGI Process Manager, and is exactly what it sounds like: a manager for PHP that’s being interpreted via FastCGI. Unlike mod_php, this means that PHP is being interpreted by FastCGI, and not directly within the Apache application.
  • UNIX socket – a mechanism that allows two processes within the same UNIX/Linux operating system to communicate with one another.

Defining threads versus processes in detail is beyond the scope of this article, but basically, a thread is much smaller and less resource-intense than a process. There are many benefits to using threads instead of full processes for smaller tasks.

2. Introduction:

Okay, so now that we have a few operational definitions in place, let’s get to the underlying problem that this article addresses. When running one or more busy websites on a server, it’s desirable to reduce the amount of processing power and memory that is needed to serve those sites. Doing so will, consequently, allow the sites to be served more efficiently, rapidly, and to more simultaneous clients. One great way to do that is to switch from the Prefork MPM to the new Event MPM in Apache 2.4. However, that requires getting rid of mod_php and switching to a PHP-FPM backend proxied via mod_proxy_fcgi. All that sounds fine and dandy, except that there have been several problems with it in the past (such as .htaccess files not being honoured, effectively passing the processed PHP file back to Apache, and making the whole system work with virtual hosts [vhosts]). Later, I will show you a method that addresses these problems, but beforehand, it might help to see some diagrams that outline these two different MPMs and their respective connections to PHP process.

The de facto way to use PHP and Apache has been with the embedded PHP processor via mod_php:

Apache Prefork MPM with mod_php embedded
Apache with the Prefork MPM and embedded mod_php
Click to enlarge

 

This new method involves proxying the PHP processing to PHP-FPM:

Apache Event MPM to PHP-FPM via mod_proxy_fcgi
Apache with the Event MPM offloading to PHP-FPM via mod_proxy_fcgi
Click to enlarge

3. The Setup:

On the majority of my servers, I use Gentoo Linux, and as such, the exact file locations or methods for each task may be different on your server(s). Despite the distro-specific differences, though, the overall configuration should be mostly the same.

3a. OS and Linux kernel:

Before jumping into the package rebuilds needed to swap Apache to the Event MPM and to use PHP-FPM, a few OS/kernel-related items should be confirmed or modified: 1) epoll support, 2) the maximum number of open file descriptors, and 3) the number of backlogged sockets.

Epoll support
The Apache Event MPM and PHP-FPM can use several different event mechanisms, but on a Linux system, it is advisable to use epoll. Even though epoll has been the default since the 2.6 branch of the Linux kernel, you should still confirm that it is available on your system. That can be done with two commands (one for kernel support, and one for glibc support):

Kernel support for epoll (replace /usr/src/linux/.config with the actual location of your kernel config):
# grep -i epoll /usr/src/linux/.config
CONFIG_EPOLL=y

glibc support for epoll
# nm -D /lib/libc.so.6 | grep -i epoll
00000000000e7dc0 T epoll_create
00000000000e7df0 T epoll_create1
00000000000e7e20 T epoll_ctl
00000000000e7a40 T epoll_pwait
00000000000e7e50 T epoll_wait

Max open file descriptors
Another aspect you will need to tune based on your specific needs is the maximum number of open file descriptors. Basically, since there will be files opened for the Apache connections, the UNIX socket to PHP-FPM, and for the PHP-FPM processes themselves, it is a good idea to have a high maximum number of open files. You can check the current kernel-imposed limit by using the following command:

# cat /proc/sys/fs/file-max
3291373

Though that number is usually very high, you also need to check the limits based on user. For sake of ease, and seeing as servers running many vhosts may frequently add and modify user accounts, I generally set the max file descriptors pretty high for *all* users.

# grep nofile /etc/security/limits.conf
# - nofile - max number of open file descriptors
* soft nofile 16384
* hard nofile 16384

The specifics of modifying the limits.conf file are outside of the scope of this article, but the above changes will set the soft and hard limits to 16,384 open files for any user on the system.

Backlogged socket connections
The last OS configuration that you might want to initially configure is the maximum number of backlogged socket connections. This number is important because we will be proxying Apache to PHP-FPM via UNIX sockets, and this value sets the maximum number of connections per socket. It should be noted that this is the maximum number per socket, so setting this kernel parameter to some outlandish value doesn’t really make a lot of sense. As you will see later in this tutorial, I create a socket for each virtual host, so unless you plan on having more than 1024 simultaneous connections to PHP-FPM per site, this value is sufficient. You can change it permanently in /etc/sysctl.conf:

# grep somaxconn /etc/sysctl.conf
net.core.somaxconn = 1024

and remember to run sysctl -p thereafter in order to make the changes active.

3b. Package rebuilds:

Now that you’ve checked (and possibly modified) some OS-related configurations, a few packages needed to be recompiled in order to change from the Apache Prefork MPM to the Event MPM and to use PHP-FPM. Here’s a list of the changes that I needed to make:

  • In /etc/portage/make.conf
    • Change the MPM to read APACHE2_MPMS="event"
    • Ensure you have at least the following APACHE2_MODULES enabled: cgid proxy proxy_fcgi
    • Per the mod_cgid documentation you should enable mod_cgid and disable mod_cgi when using the Event MPM
  • In /etc/portage/package.use
    • Apache: www-servers/apache threads
    • PHP must include at least: dev-lang/php cgi fpm threads
      • Note that you do not need to have the ‘apache2’ USE flag enabled for PHP since you’re not using Apache’s PHP module
      • However, you will still need to have the MIME types and the DirectoryIndex directives set (see subsection 3d below)
    • Eselect for PHP must have: app-eselect/eselect-php fpm

Now that those changes have been made, and you’ve recompiled the appropriate packages (Apache, and PHP), it’s time to start configuring the packages before reloading them. In Gentoo, these changes won’t take effect until you restart the running instances. For me, that meant that I could compile everything ahead of time, and then make the configuration changes listed in the remaining 3.x subsections before restarting. That procedure allowed for very little downtime!

3c. Apache:

Within the configurations for the main Apache application, there are several changes that need to be made due to switching to a threaded version with the Event MPM. The first three bullet points for httpd.conf listed below relate to the proxy modules, and the fourth bullet point relates to the directives specific to the Event MPM. For more information about the MPM directives, see Apache’s documentation.

  • In /etc/apache2/httpd.conf
    • The proxy and proxy_fcgi modules need to be loaded:
      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
    • Comment out LoadModule cgi_module modules/mod_cgi.so so it does not get loaded
    • Replace that with the one for mod_cgid: LoadModule cgid_module modules/mod_cgid.so
    • Update the MPM portion for Event:
      • <IfModule event.c>
        StartServers 2
        ServerLimit 16
        MinSpareThreads 75
        MaxSpareThreads 250
        ThreadsPerChild 25
        MaxRequestWorkers 400
        MaxConnectionsPerChild 10000
        </IfModule>

You will also need to make sure that Apache loads certain modules at runtime. In Gentoo, that configuration is done in /etc/conf.d/apache, and needs to contain at least the two modules listed below (note that your configuration will likely have other modules loaded as well):

  • In /etc/conf.d/apache
    • APACHE2_OPTS="-D PHP5 -D PROXY"

3d. Apache PHP module:

The last change that needs to be made for Apache is to the configuration file for how it handles PHP processing (in Gentoo, this configuration is found in /etc/apache2/modules.d/70_mod_php5.conf. I would suggest making a backup of the existing file, and making changes to a copy so that it can be easily reverted, if necessary.

70_mod_php5.conf BEFORE changes
<IfDefine PHP5>
  <IfModule !mod_php5.c>
    LoadModule php5_module modules/libphp5.so
  </IfModule>
  <FilesMatch "\.(php|php5|phtml)$">
    SetHandler application/x-httpd-php
  </FilesMatch>
  <FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source
  </FilesMatch>

  DirectoryIndex index.php index.phtml
</IfDefine>

70_mod_php5.conf AFTER changes
<IfDefine PHP5>
  ## Define FilesMatch in each vhost
  <IfModule mod_mime.c>
    AddHandler application/x-httpd-php .php .php5 .phtml
    AddHandler application/x-httpd-php-source .phps
  </IfModule>

  DirectoryIndex index.php index.phtml
</IfDefine>

Basically, you’re getting rid of the LoadModule reference for mod_php, and all the FileMatch directives. Now, if you’re not using virtual hosts, you can put the FilesMatch and Proxy directives from the vhosts section (3e) below in your main PHP module configuration. That being said, the point of this article is to set up everything for use with vhosts, and as such, the module should look like the example immediately above.

As mentioned in section 3b above, the reason you still need the 70_mod_php5.conf at all is so that you can definite the MIME types for PHP and the DirectoryIndex. If you would rather not have this file present, (since you’re not really using mod_php), you can just place these directives (the AddHandler and DirectoryIndex ones) in your main Apache configuration—your choice.

3e. vhosts:

For each vhost, I would recommend creating a user and corresponding group. How you manage local users and groups on your server is beyond the scope of this document. For this example, though, we’re going to have site1.com and site2.com (very creative, I know 😎 ). To keep it simple, the corresponding users will be ‘site1’ and ‘site2’, respectively. For each site, you will have a separate vhost configuration (and again, how you manage your vhosts is beyond the scope of this document). Within each vhost, the big change that you need to make is to add the FilesMatch directive. Here’s the generic template for the vhost configuration additions:

Generic template for vhost configs
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/$pool.sock|fcgi://$pool/"
</FilesMatch>

In that template, you will replace $pool with the name of each PHP-FPM pool, which will be explained in the next subsection (3f). In my opinion, it is easiest to keep the name of the pool the same as the user assigned to each site, but you’re free to change those naming conventions so that they make sense to you. Based on my site1.com / site2.com example, the vhost configuration additions would be:

site1.com vhost config
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/site1.sock|fcgi://site1/"
</FilesMatch>

site2.com vhost config
<FilesMatch "\.php$">
  SetHandler "proxy:unix:///var/run/php-fpm/site2.sock|fcgi://site2/"
</FilesMatch>

3f. PHP-FPM:

Since you re-compiled PHP with FPM support (in step 3b), you need to actually start the new process with /etc/init.d/php-fpm start and add it to the default runlevel so that it starts automatically when the server boots (note that this command varies based on your init system):

rc-update add php-fpm default (Gentoo with OpenRC)

Now it’s time to actually configure PHP-FPM via the php-fpm.conf file, which, in Gentoo, is located at /etc/php/fpm-php$VERSION/php-fpm.conf. There are two sections to this configuration file: 1) global directives, which apply to all PHP-FPM instances, and 2) pool directives, which can be changed for each pool (e.g. each vhost). For various non-mandatory options, please see the full list of PHP-FPM directives.

At the top of php-fpm.conf are the global directives, which are:
[global]
error_log = /var/log/php-fpm.log
events.mechanism = epoll
emergency_restart_threshold = 0

These directives should be fairly self-explanatory, but here’s a quick summary:

  • [global] – The intro block stating that the following directives are global and not pool-specific
  • error_log – The file to which PHP-FPM will log errors and other notices
  • events.mechanism – What should PHP-FPM use to process events (relates to the epoll portion in subsection 3a)
  • emergency_restart_threshold – How many PHP-FPM children must die improperly before it automatically restarts (0 means the threshold is disabled)

Thereafter are the pool directives, for which I follow this template, with one pool for each vhost:
;; $SITE.$TLD
[$USER]
listen = /var/run/php-fpm/$USER.sock
listen.owner = $USER
listen.group = apache
listen.mode = 0660
user = $USER
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

It may look a bit intimidating at first, but only the $SITE, $TLD, and $USER variables change based on your vhosts and corresponding users. When the template is used for the site1.com / site2.com example, the pools look like:

;; site1.com
[site1]
listen = /var/run/php-fpm/site1.sock
listen.owner = site1
listen.group = apache
listen.mode = 0660
user = site1
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

;; site2.com
[site2]
listen = /var/run/php-fpm/site2.sock
listen.owner = site2
listen.group = apache
listen.mode = 0660
user = site2
group = apache
pm = dynamic
pm.start_servers = 3
pm.max_children = 100
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.max_requests = 10000
request_terminate_timeout = 300

Since the UNIX sockets are created in /var/run/php-fpm/, you need to make sure that that directory exists and is owned by root. The listen.owner and listen.group directives will make the actual UNIX socket for the pool owned by $USER:apache, and will make sure that the apache group can write to it (which is necessary).

The following directives are ones that you will want to change based on site traffic and server resources:

  • pm.start_servers – The number of PHP-FPM children that should be spawned automatically
  • pm.max_children – The maximum number of children allowed (connection limit)
  • pm.min_spare_servers – The minimum number of spare idle PHP-FPM servers to have available
  • pm.max_spare_servers – The maximum number of spare idle PHP-FPM servers to have available
  • pm.max_requests – Maximum number of requests each child should handle before re-spawning
  • pm.request_terminate_timeout – Maximum amount of time to process a request (similar to max_execution_time in php.ini

These directives should all be adjusted based on the needs of each site. For instance, a really busy site with many, many, simultaneous PHP connections may need 3 servers each with 100 children, whilst a low-traffic site may only only need 1 server with 10 children. The thing to remember is that those children are only active whilst actually processing PHP, which could be a very short amount of time (possibly measured in milliseconds). Contrapositively, setting the numbers too high for the server to handle (in terms of available memory and processor) will result in poor performance when sites are under load. As with anything, tuning the pool requirements will take time and analysis to get it right.

Remember to reload (or restart) PHP-FPM after making any changes to its configuration (global or pool-based):

/etc/init.d/php-fpm reload

4. Verification:

After you have configured Apache, your vhosts, PHP, PHP-FPM, and the other components mentioned throughout this article, you will need to restart them all (just for sake of cleanness). To verify that things are as they should be, you can simply browse to one of the sites configured in your vhosts and make sure that it functions as intended. You can also verify some of the individual components with the following commands:

Apache supports threads, and is using the Event MPM
# apache2 -V | grep 'Server MPM\|threaded'
Server MPM:     event
  threaded:     yes (fixed thread count)

Apache / vhost syntax
# apache2ctl configtest
 * Checking apache2 configuration ...     [ ok ]

5. Troubleshooting:

So you followed the directions here, and it didn’t go flawlessly?! Hark! 😮 In all seriousness, there are several potential points of failure for setting up Apache to communicate with PHP-FPM via mod_proxy_fcgi, but fortunately, there are some methods for troubleshooting (one of the beauties of Linux and other UNIX derivatives is logging).

By default, mod_proxy and mod_proxy_fcgi will report errors to the log that you have specified via the ErrorLog directive, either in your overall Apache configuration, or within each vhost. Those logs serve as a good starting point for tracking down problems. For instance, before I had appropriately set the UNIX socket permission options in php-fpm.conf, I noticed these errors:

[Wed Jul 08 00:03:26.717538 2015] [proxy:error] [pid 2582:tid 140159555143424] (13)Permission denied: AH02454: FCGI: attempt to connect to Unix domain socket /var/run/php-fpm/site1.sock (*) failed
[Wed Jul 08 00:03:26.717548 2015] [proxy_fcgi:error] [pid 2582:tid 140159555143424] [client 52350] AH01079: failed to make connection to backend: httpd-UDS

That error in bold text indicated that Apache didn’t have permission to write to the UNIX socket. Setting the permissions accordingly—see the listen.owner and listen.group directives portion of subsection 3f—fixed the problem.

Say that the errors don’t provide detailed enough information, though. You can set the LogLevel directive (again, either in your overall Apache configuration, or per vhost) to substantially increase the verbosity—to debug or even trace levels. You can even change the log level just for certain modules, so that you don’t get bogged down with information that is irrelevant to your particular error. For instance:

LogLevel warn proxy:debug proxy_fcgi:debug

will set the overall LogLevel to “warn,” but the LogLevel for mod_proxy and mod_proxy_fcgi to “debug,” in order to make them more verbose.

6. Conclusion:

If you’ve stuck with me throughout this gigantic post, you’re either really interested in systems engineering and the web stack, or you may have some borderline masochistic tendencies. 😛 I hope that you have found the article helpful in setting up PHP to proxy PHP interpretation to PHP-FPM via mod_proxy_fcgi, and that you’re able to see the benefits of this type of offloading. Not only does this method of processing PHP files free up resources so that Apache can serve more simultaneous connections with fewer resources, but it also more closely adheres to the UNIX philosophy of having applications perform only one task, and do so in the best way possible.

If you have any questions, comments, concerns, or suggestions, please feel free to leave a comment.

Cheers,
Zach

Aug 03 2015

Syncopation red wine blend from Mike Ward on Wine

Syncopation red wine blend - Mike Ward on Wine - Augusta, MOUPDATE: See my review on the new 2015 vintage of Syncopation, including the Acoustic White blend.

This past weekend, I was visiting my good friends at The Wine Barrel for their weekly wine tasting, and I got a neat surprise when I was there. Certified Sommelier, Mike Ward of Ward on Wine was there with a new project of his called Syncopation. Syncopation is a private-label red wine blend produced in Augusta, MO, which, if you didn’t know, was the first AVA recognised by the United States Federal Government in 1980 (beating out Napa).

Having been to some of Mike’s classes (including ones about two of my favourite countries for wine [Italy and Spain]), I was confident that any wine he would stand behind would be a worthwhile investment. Though I’m not usually partial to wines from Missouri, the way that he described Syncopation as being “medium bodied with some nice red fruits and a little spice” was enticing to me. Before getting to my personal impressions and review of Syncopation Rhythmic Red blend, here are some interesting notes about it:

  • It is a blend of Chambourcin, Vidal blanc, Seyval blanc, and Traminette
    • Chambourcin is really a red grape, whilst the other three are white grapes
    • Chambourcin can be made so that it is dry or semi-sweet
    • Vidal blanc is related to Trebbiano, which is a grape noted for wines from Italy and France (where it is more commonly known as Ugni blanc)

Mike indicated to me that this particular wine is best enjoyed at a slightly chilled temperature (16-18°C / 60-65°F), but that I should try it right out of the bottle beforehand, and make my own assessments. I will agree with him that it is better with a light chill on it, and after it has decanted for 20-30 minutes. Since I did try it both ways, I will separate my tasting notes into two corresponding sections.

Right out of the bottle
When I first tried Syncopation, I noticed a slight sweetness to it, which is the signature of some of the white blends from St. James Winery and others in Missouri. It was forward with a burst of freshly picked strawberries, and had delicate notes of Victoria plums in the mid-palate. It was light-to-medium bodied, and had an almost effervescent mouthfeel, though there were no bubbles at all.

Chilled accordingly
After my bottle of Syncopation had chilled to the recommended temperature, I tried it again. To me, the differences were night and day! As expected, the slight chill intensified the flavours of strawberry and plum that I had originally noted, making them more bold and readily recognisable. The effervescence that I mentioned was no longer present, and the mouthfeel had tightened to a more medium-body. Most impressive to me was how the chilling and decanting brought some underlying subtleties to the surface. I really enjoyed some notes of cherry cola, white pepper, and most prominently, coastal sage scrub.

Mike Ward on Wine and Zach - first sale of Syncopation Rhythmic Red Blend
Mike Ward and Zach – First sale of Syncopation Red Wine
Click to enlarge

Overall impressions
As someone who enjoys the intense wines of Priorat, Spain and yet the subtleties of a Brunello di Montelcino, I found Sycnopation to be a bit light for my preference. That being said, it is an approachable red wine for sweet and semi-sweet white wine lovers, yet tannic enough to appease staunch red drinkers. To me, Syncopation lives up to its name in that it is rhythmic and flowing. It also captures the joys of summer, and is able to—with its beautiful notes of strawberry and white-fleshed plum—cut through the ofttimes oppressive humidity that can accompany a Saint Louis August. If you think you know what Missouri wines are “all about,” or if you just would like an unassuming light-to-medium red blend that is new and exciting, I urge you to give Syncopation a try!

Cheers,
Zach