Earlier today Zoho, a leading cloud services provider whose CRM solution is known as a solid competitor to Salesforce.com, went off the air.  The root cause it turns out was a power outage at their colocation provider’s data center.  Their colo provider, Equinix, is considered to be a top-tier provider, and while power at the data center has been restored, Zoho is still down hours later trying to fix all the data corruption from what was effectively pulling the power cords out of the back of the servers while the servers were still running.

Now, Zoho has several million users, including us, so fixing data corruption of that magnitude is not like letting Windows chkdsk just run for a few minutes after the server is rebooted. We’ll have to wait to see what the final outcome is, and for how long Zoho CRM (and SugarCRM and another 214 customers Equinix claims to host at that data center) remain down.

We suffered the same fate a few years ago at our former colocation host.  That and other issues caused us to move to a new colocation facility.  What happened at our former colocation host was that there was a power outage, the data center UPS (uninterruptible power supply) kicked in, and the system waited for the generator to start.  Only the generator didn’t start, and the UPS system had only a few minutes of juice in their batteries, so every server in the data center crashed, quite hard.  Fortunately, we had plans in place so we were able to recover quickly.

When we did a new colocation facility bakeoff, one of the detailed questions we asked was what happens if the power goes out and the generator fails to start?  Most data centers told us things like “We test the generator weekly! That won’t happen!” (which is what our former data center provider told us as well). Well, guess what?  You-know-what does happen periodically.

At the end of the day, we chose BayRing Communications, a New Hampshire-based phone company with two data centers at the old Pease Air Force base.  When we asked that same question of them, they laughed, literally, and said that in their experience gear fails all the time and so one needs to be prepared.  In their case, they bought a lot of batteries for their UPSs. When the power goes out, their UPS can run everything for several hours – plenty of time to either fix the generator or get a portable generator trucked in and hooked up.  They reminded us that, as a phone company, they get in big trouble if things like 911 don’t work for any length of time.

Indeed, at the end of the due diligence, we understood in more intimate detail what “carrier-grade” really means. And why, if you are running your own and hosting your clients’ mission-critical applications (like electronic health records, and email for regulated companies for example), “carrier-grade” has to be the minimum standard.

Does that cost more? More than some and less than others.

Will we survive without access to our CRM application through Zoho for a few hours? Sure. For a few days would be a real problem though.

At the end of the day, the takeaway here is that, whether you are taking care of a few dozen customers or a few million, when you choose a data center provider you really need to do your due diligence carefully.  Something clearly went horribly wrong at Equinix, and as of this writing, though power has been restored for a few hours, they haven’t disclosed the root cause.  We’ll have to wait and see…

If you have mission-critical applications and you have concerns about their hosting, we’d be happy to help you through a due diligence process that we organized for ourselves and our clients who host with us. Just give us a call at (207) 772-5678.

Mark

We recently came a cross an interesting study released by international security firm CSIS, which concludes that “…as much as 99.8 % of all virus/malware infections caused by commercial exploit kits are a direct result of the lack of updating five specific software packages.”

According to CSIS, “Up to 85 % of all virus infections occur as a result of drive-by attacks automated via commercial exploit kits.”

Commercial exploit kits are used both by the Bad Guys as well as by legitimate security shops to perform penetration testing, security scans and other tests appropriate for regulated companies and unregulated companies with valuable intellectual property to protect. Like guns, how these tools are used is up to the person with their finger on the trigger.

We have always known that patching your systems is very important.  Unfortunately, even though Windows Update now patches Office and other Microsoft software in addition to the Windows operating system itself, that’s not enough.  And now CSIS has the facts to prove it.

The five software packages that accounted for that 99.8% of all infections as reported by CSIS comprise:

1. Java JRE (which includes the browser plugin)
2. Adobe Reader
3. Adobe Acrobat
4. Adobe Flash and,
5. Microsoft Internet Explorer

While Internet Explorer gets patched via Windows Update, the other software packages have their own update system that prompts the user to update each software package individually.

Regrettably, in our experience way too many end-users ignore and click away those update warnings.  Further, in in more locked-down corporate environments, end-users often do not have sufficient rights to install software updates and patches; those updates and patches are pushed out to the end users’ machines via centralized system management software.  When the company’s system administrators do not push those patches out promptly enough, or the end users click away and defer updating those packages, a significant exposure is created.

And once one machine on a network is infected, many more are often subsequently infected.

So what does this all mean?

Well, first… this report reaffirms the importance of prompt patching.  Second, it documents that of the five top exposures, only one of them (Internet Explorer) is patched via Windows Update, so just turning on Windows Auto Update and thinking you are protected is at best fatuous.

At the end of the day, a comprehensive patching process is required; the proper execution of which is someone’s key responsibility.

If you have concerns about your company’s patching processes, please give us a call (207) 615-1529.

Mark

CIO

P.S.  You can read a summary of the study here.

We recently reviewed a Cloud Computing survey published by Cloud.com and it got us thinking.  Cloud.com is in a terrific position to have their finger on the pulse of the overall market; they make platform-independent Cloud Management software that in one console can mange multiple cloud deployments deployed on Amazon, VMware, XenServer etc.  They were recently purchased by Citrix, who have a solid track record of supplying both Open Source and Proprietary software. (Most larger cloud stacks, like Amazon, are built on Citrix’s Open Source version of XenServer.)

The survey pointed out that workloads currently on clouds are comprised primarily of web sites, shared data storage,  backups and prototyping/sandbox.  Although most companies nowadays have adopted virtualization for line-of-business applications, the survey pointed out that very few of these of line-of-business have been migrated to the cloud.

At first, that struck me as odd since the greatest benefits of cloud computing come with cloud-ified (is that a word?) line-of-business applications.  And then the brick of realization hit me: Doh!  Most cloud environments aren’t ready to host line-of-business applications. Most line-of-business applications rely on high-performance databases (“fast disk I/O” in techno-speak), and most cloud providers have pretty slow disks — it’s how they keep prices down.

We ourselves spent eight months trying to get out of the hardware business, unsuccessfully.  Our Zimbra hosting farm hardware was approaching end-of-life, we had multiple clients who wanted to break the expensive cycle of premises-based hosting and so we had a raft of servers ready to be cloud hosted.

What we found was that the inexpensive cloud hosting providers’ infrastructure had nowhere-near-fast-enough disk I/O, and even very expensive private cloud hosting providers (at least the ones we talked to at the time) with very fast disk I/O had one or more “gotchas” that precluded us from going with them.

There are a lot of moving parts to successful cloud hosting. Not only do you have to be an expert at virtualization and eliminating single points of failure cost-effectively, but you have to be cognizant that most security standards are only a starting point. The most difficult challenge in moving clients to the cloud however is education/politics.  Cloud computing done well often requires business process improvements.

And then the second Brick ‘O Realization hit us (this was starting to hurt) that we have been doing virtualization for regulated companies, educating senior management and managing staff’s expectations for years.

So, we built our own private cloud, for our hosted Zimbra farm and for our clients. Architected explicitly for those database-intensive line-of-business applications requiring fast I/O. With 24 x 7 x 365 human client service. And priced appropriately, which is to say higher than Amazon (but not by as much as you think) and less than dedicated server hosting from the majors.

One client who runs SAS for healthcare analytics tested our private cloud and found it ran jobs anywhere from four to 10 times faster than their current blend of physical and virtual server infrastructure. Our own Zimbra hosting farm runs noticeably faster as well (and we do not overload our mailbox servers with the maximum amount of mailboxes either).

So in a nutshell, we think the big upcoming wave in Cloud Computing is migrating mission-critical, database-dependent line-of-business applications to private cloud service providers who “get” security, regulated environments and who can be relied on to manage, carefully and successfully, what can too often be a difficult transition to cloud computing.

And that’s why, after we had a chance to think about it, the survey from Cloud.com made sense and validated the investment we made in, and position of, our own private cloud environment.

If you are as scared of cloud computing as we were, give us a call at (207) 772-5678. You might rightfully decide that cloud computing is not for your company, but you might find out some things you didn’t expect.

Safe computing,

Mark, CIO

Reliable Networks Announces Private Cloud and Educational Series

Holiday Inn by the Bay – Wednesday, June 29^th 5:00pm – 6:00pm

PORTLAND, ME — June 13, 2011 — Reliable Networks, a network engineering and hosted services firm based in Portland, ME today announced its Private Cloud and Educational Series to help companies evaluate whether cloud hosting is at all appropriate for their business.

After nearly a decade of architecting and deploying private clouds for select clients, Reliable Networks has expanded the same successful formula to create the most secure and reliable cloud to hover over Maine.

Public versus Private Clouds

Outages and data breaches at large public cloud and service providers like Amazon EC2, Yahoo, Microsoft 360 and Gmail are reported with increasing frequency.  Niche private cloud providers like Reliable Networks however have been quietly providing secure, reliable services to clients for years.

Appropriateness of Cloud Hosting

Is the Cloud right for all businesses? Not necessarily. Businesses must evaluate whether cloud hosting is appropriate before even considering which type of cloud hosting is optimal, public, private, or hybrid. Reliable Networks has conducted a due diligence program to consider whether outsourcing this service would be in its clients’ best interests.  “Regrettably,” reports Reliable Networks President L. Mark Stone, “all of the private cloud providers we investigated had one or more areas of concern that made their offering inappropriate for our clients’ needs and for our own suite of hosted services.”

Cloud Hosting Educational Series

The temptation to move to the cloud is great, given the myraid benefits including cost savings, higher reliability and more efficient workflow processes.  “But as our own due diligence exposed, the road to Nirvana has a few IEDs buried along the shoulder.”  added Stone. “So we thought we should share the results of our cloud hosting due diligence with the community, to help others make better decisions about whether to, and if so, how to, host in the cloud.”

The first session in this Cloud Hosting Educational Series will be held on Wednesday, June 29^th from  5:00pm to 6:00pm at the Holiday Inn By The Bay in Portland.  Pre-registration is required (no at-event signups can be admitted) at www.reliablenetworks.com/events and complimentary refreshments, sushi, beer and wine will be served.

For more information:

Kristin Przybysz, (207) 772-5678, kristin@reliablenetworks.com

Featured in this month’s MaineAhead Magazine is an article we wrote about the tragic risks and consequences associated with a data breach. Reliable Networks founder, L. Mark Stone, recounts a speaking engagement at an October health care conference, MIMS2010. Physicians there were alarmed to learn that neither general liability insurance nor malpractice insurance typically covers a medical data breach, even if they were HIPAA compliant at the time of the breach.

The reasons for this coverage gap are several:  First, property and casualty insurance policies are written to cover tangible items and data isn’t tangible. Medical malpractice insurance policies don’t consider a data breach a medical error and so don’t usually cover the costs from data breaches. Even General Liability policies rarely include data breaches in the specific list of liabilities covered.

Worse, breaches are expensive! The estimated cost for a data breach spans $220 – $330 per record. Consider a primary care physician with a panel (i.e. patient base) of 4,000 patients. The practitioner’s cash out-of-pocket costs to remedy a typical data breach could exceed $1.0 million. Almost all states mandate some form of data breach reporting, and a quick search on DatalossDB.org shows health care providers are reporting data breaches frequently. Fail to report appropriately and your out-of-pocket costs go up; Stanford was recently fined $250,000 for failing to report a breach on a timely basis.

As businesses convert to electronic records or migrate to the cloud, increasingly more insurance companies offer cyber liability and data breach insurances. Rates vary depending on the risk within the practice. But it’s not easy obtaining cyber liability and data breach insurance. The vetting process is very thorough and time-consuming. We tell clients that the process is not unlike going through an actual security review — not a bad thing to do in any event since being “compliant” doesn’t necessarily mean you are “secure”!

The good news is that, with a combination of properly trained personnel and a secure network (not anywhere as expensive a proposition as you might think), any company can reduce the likelihood of these tragic and unexpected costs.

If you would like to see how your network security configurations compare with others, please feel free to call us at 207-772-5678.

Kristin Przybysz
Business Development

The wonderful standard “Take the A Train” is a terrific example of early social media marketing.  Although originally crafted as a result of directions Duke Ellington gave Billy Strayhorn to get to his house in Harlem, the song was later heavily promoted by The Cotton Club, also located in Harlem, and accessible by… the A train.  What better than to have a popular song let your prospective patrons know how to get to your club, long before such things like Google Maps or Siri were around?

Now that insurance companies and other payers have cottoned on (yes, pun intended, sorry!) to the fact that preventative medicine lowers health care costs, primary care and other specialty providers are starting to dip their toes in the waters of better managing their patient acquisition methodologies.  Practices whose revenues rely more on elective procedures have traditionally been leaders in the medical marketing space, and as we all know, the Internet is where marketing lives these days.

In that regard, I was very pleased to be invited to speak at the upcoming Medical Internet Marketing Symposium being held in Boston on October 29-30.  Their website is http://www.mims2010.com.

The symposium addresses not only core Internet marketing challenges, but also is intended to help attendees with related issues and opportunities: e.g. how iPad usage in medical practices creates tremendous efficiencies; what to look for when evaluating patient portals; Google’s perspectives on patients’ use of medical information; good website design; computer security and data breaches, and more.

Targeted specifically for medical practices, the attendees already registered comprise a mix of physicians, practice managers and other senior, non-technical medical executive staff.  So, while many of the presentations cover technical topics, the goal is to empower attendees to plan and execute an Internet marketing strategy appropriate for their practice, and not how to code middleware between Centricity and a third-party web portal (although we can certainly do that for you if you need us to).

I hope you will consider joining us at MIMS 2010 later this month!  (You can register directly on the site.)

All the best,
Mark
CIO

P.S. Please feel free to contact me directly if you have any questions about this conference.

(“Borks” is a technical term meaning “really messed up” in somewhat less polite terms…)

A recent Microsoft patch MS10-015, which requires a reboot to complete the install, is reported to be causing a number of Microsoft servers and workstations to fail to reboot at all; the reboot ends with the infamous “Blue Screen of Death” and renders the system unusable.

Putting aside for a moment that this patch fixes a security hole Microsoft has known about for seventeen years, we think this incident highlights the need for a multi-layered approach to security in the first instance.

Microsoft is claiming that a number of systems experiencing the Blue Screen of Death are doing so because the systems were already compromised. So, right away, that tells you that even applying patches quickly isn’t enough to keep systems safe.

Further, the risk with applying patches immediately when they are released is that you will bork your system. It doesn’t happen often, but when you consider how expensive downtime really is, even once every few years is very expensive. (As I write this post, I see that Microsoft has pulled the patch to avoid borking additional systems.)

Solid network perimeter protection has been a staple of our best practices for years. Smaller clients sometimes balk initially at spending hundreds of dollars for an enterprise-grade firewall, but these devices represent cheap insurance at worst and in many cases generate a positive return on investment.

End-user education and “safe-browsing” policies are also required to avoid security breaches. Malware these days (as we have blogged previously) is increasingly sophisticated and insidious. Firewalls and anti-virus/malware software will always be a few steps behind.

So, when you have good perimeter protection and careful, educated end-users, you have the luxury of time in which to evaluate new patches as they are issued. The benefits are you stay safer all the time, and reduce the risk of borking your production systems.

If you need help with your company’s patch management and security posture, please give us a call at (207) 772-5678.

Hope that helps,
Mark
CIO

Just a few short years ago most virus writers were amateurs trying to trash your PC, just because they could. Nowadays, the “malware” industry is very professional, dominated by organized crime, whose products keep your PC running to enable sensitive keystroke/data logging (think Hannaford, TJ MAXX, etc.), to send spam, or to try to extort money from you with phony “Your computer is infected! Click here to fix!” scams.

The anti-virus software companies have raised the bar by bundling more protections into their traditional anti-virus products, in many cases releasing them as new products: Symantec Endpoint Protection for example has replaced Symantec Anti-Virus Corporate Edition. And although you can still buy products labeled as “anti-virus” from the majors, sales of these limited-use products have declined considerably of late.

As these new protection products have become more complex, sophisticated and bloated, their performance impact on older PCs has become more noticeable. Bargain PCs purchased as recently as two years ago can be too slow to be used efficiently with these full-suite protection products installed. Furthermore, the protection products are by nature always playing “catch-up” with the bad guys, so we have seen some companies forgo desktop protection software altogether in favor of strong network perimeter protection combined with policies limiting Internet access and prohibiting employees from bringing in cdroms, USB drives, outside laptops etc. into the office. (The more powerful servers still have protection software installed however.)

That’s one way to do it, but many companies can’t manage the politics associated with limiting or preventing employees from browsing the Internet. And some companies, like ad agencies and web developers, can’t really be restricted at all.

Further, we have seen a lot of malware that these protection products simply can’t protect against, because the malware looks and acts like legitimate software.

Switching to a Mac or a Linux PC can help, but these devices can become “carriers” for malware, bringing a whole host of new challenges. And most security pundits believe that as Macs and Linux PCs become more popular, it will only be a matter of time before malware for these machines starts appearing as well.

So if these big protection products can’t save us from ourselves reliably, what can be done?

Well, here is our list of the top four ways you can protect yourself.

1. Slow Down! We have seen malware come in via email looking like Hallmark e-cards, IRS W-2 form updates, PayPal and bank account alerts, etc. If a friend’s PC gets an infection, you will get an email from your friend, and the web link or attachment that looks so enticing (if not workplace safe) will be your downfall. So, before you click on anything, take a moment to scan it with your own brainpower and a skeptical eye.

2. Be Careful Where You Stick Your Browser. The San Fransisco bath house analogy notwithstanding, the majority of malware infections these days are installed via a web link. Staying away from those web sites you know you shouldn’t be frequenting anyway is a good start, but keep in mind that malware writers are very clever. They do things like buy ads on legitimate web sites to distribute their wares, so just because you are on cnn.com doesn’t mean you can click anywhere safely 100% of the time. When you get a popup or other prompt to take an action you weren’t expecting, apply Rule #1 and slow down before doing anything.

3. Be Proactive and Scan Your PC. Whether you use malware protection software or not, periodically being proactive and running scans on your PC at least once a month is a good thing. We like Malware Bytes a lot, but our favorite tool de jour is Combo Fix, available as of this writing here. Be careful when you search for these tools; the malware folks have bought look-alike domains and lots of Google AdWords! We have seen several folks with a minor infection wind up with a totally borked workstation because the web link they thought was malware removal software from the good guys was actually more malware from the bad guys. Remember Rules #1 and #2?

4. Keep Your PC Patched. The majority of patches coming out of Microsoft are security, not bug fixes. Making sure your PC is regularly updated is key. If you are running non-Microsoft products, like Adobe Acrobat Reader, Apple’s Quicktime, etc. you want to be sure those products are kept up to date as well. Acrobat products this week are being blasted in the trade press because the Javascript code in the product has been a valuable attack vector for malware developers. Adobe can’t or won’t “fix” this because the same Javascript code is used for filling in PDF forms, and Adobe doesn’t want to hinder that functionality.

So be safe out there! And if you have questions or get yourself in trouble, we are here to help. Don’t send us an infected email though, just give us a call at (207) 772-5678.

All the best,
Mark
CIO