Over that time, as technology has changed, so has the makeup of our clients – but not what we accomplish for them. David Bowie’s song “Changes” instructs: “Turn and face the strange… Changes.” Certainly that is apt advice for our clients trying to maximize benefits from their technology investments.

So I thought it would be helpful to let everyone know some interesting news and recap a few core principles that comprise who we are and how we spend our days in this strange world of technology.

Core Values
The most important thing to us is our clients, who:

1. Are smart. Wicked smart. So smart they force us to be our best every day.
2. Have as a core value leveraging technology to improve communication and collaboration between employees, customers, vendors, community, politicians, etc. – everyone whom they “touch” to get business done.
3. May, or may not (nor care to) understand all the technical bits of a particular solution, but require us to know our stuff and be free from conflict in recommending and implementing solutions – which is why we do not act as a reseller.
4. Demand an open, constructive dialog with their technology consultant and managed services provider to collaboratively architect, deploy, maintain and eventually life cycle mission-critical core systems on which the health of the business depends. One client called us the “…constructive thorn in our side.”. High praise that confirms we are successful in helping our clients obtain an appropriate balance between the sometimes-conflicting arts of business-expedient and technical-grace.

Most of our newer clients (increasingly who are out-of-state) now rely on us for our private cloud expertise to architect, deploy and in many cases host their mission-critical applications. Whether that application be email-based (Zimbra or Exchange), an EHR/EMR, LAMP stack, database, image-rendering, ERP, accounting, or whatever, we have found that each application has its own peculiarities but a significant amount of commonality with others. Our trademarked “Uptime. All the time.”® is the standard our clients require and what we strive to provide.

Supporting mission-critical applications alone is hard; providing a first-class private cloud in which to host them is harder. We deployed our own private cloud last summer after first trying to outsource hosting to a separate private cloud provider – and finding no vendor who met our standards. And that gets us to some interesting news:

News
New SAN Storage. Already we are about to outgrow our private cloud’s SAN storage infrastructure (about a year sooner than our optimistic forecast) and will be placing an order for new storage within the week. We have also added to the compute head side of the farm three times since going live last summer.

New Certification. Increasingly we find we are sought out by companies in regulated industries for whom an SSAE16 SOC 2 Type II is mandatory. Last year we successfully completed a third-party HIPAA/HITECH Act review and this year we engaged Moody, Famiglietti and Andronico, LLP in Massachusetts to help us get our SOC2 Type II report before the end of this year. We have been following ITIL processes for some time, so we have had no surprises getting ready for the actual SSAE16 audit.

New Insurance. We have had $2.0 million of technology errors and omissions insurance for some years now, but we recently doubled our aggregate commercial general liability to $4.0 million to better meet the needs of some of our newer clients.

New Offices. To provide room for our continued, albeit carefully measured, expansion, we will be moving offices early this summer and are evaluating our options currently.

Client Impacts
Hardly any. Other than an address change, we expect all of the above news items will help us to continue to provide the exceptional standard of work we demand of ourselves and which our clients demand of us.

The Future
For the majority of our clients, a near or fully virtualized SAN-backed application hosting environment deployed on premises or in a private cloud is the common theme. Consider the benefits:

1. Desktop security concerns and support costs go down significantly when Desktops are virtualized.
2. Server maintenance is simplified, and patch testing risk, backup, disaster recovery and business continuity costs are reduced.
3. Storage management is similarly simplified and total storage costs reduced (over DASD).
4. Application and platform migrations (e.g. Exchange 2003 on Windows Server 2003 to Exchange 2010 on Windows Server 2008) are similarly simplified and costs reduced.

In the same way that electronic medical records have created new kinds of medical errors, getting your applications properly hosted either in a private cloud or on premises in a SAN-backed virtualized infrastructure creates new kinds of exposures. By way of example, your virtualized server may move all by itself across physical hosts. Strange? Sure (thank you Mr. Bowie). And boy does it ever change your disaster recovery plans.

That’s where we can help. To learn more, give us a call (our phone number is not changing!) at (207) 772-5678.

All the best,
Mark
CIO

Recently, Novell released the latest Service Pack (SP2) for SuSE Linux Enterprise Server 11. We urge great caution before upgrading your existing production SLES servers to this new Service Pack.

Background

Under SLES 9 and 10, it was very safe to apply newly released Service Packs to production systems.  SLES 11 however broke with that trend (twice now) in a fairly major way.

SLES 11 SP1 introduced a new version of glibc (a major system library) that caused havoc for developers. Take Zimbra for example (we are a Zimbra hosting partner…); up to certain versions, Zimbra would run on SLES 11 only, but not on SP1.  After a certain Zimbra version, Zimbra would not run on SLES 11, just SLES 11 SP1. Ugh.

SLES 11 SP2 now introduces the 3.0 Linux kernel; historically SuSE never upgraded the major kernel version during the lifetime of a SLES product, though it did backport bug fixes and security updates from later kernel versions. This backporting of features, fixes and security patches provided the best of both worlds – a really stable enterprise Linux platform with a secure kernel containing modern features.  Indeed, IBM’s Linux distro of choice for Linux virtualized on their mainframes is SLES; RedHat is relegated primarily to lower-end x86 platforms by IBM.

Developers Won’t Support It Now

We asked Zimbra what their plans were for supporting SLES 11 SP2 and were told that they will start supporting SP2 only on Zimbra 8, due to be released later this year.  In other words, don’t upgrade SLES 11 to SP2 on any existing Zimbra system.

Now What?

What happens if Zimbra delays the release of version 8 past the date that Novell provides fixes for SP1?  The short answer is we are stuck with a system which could contain security exposures and bugs at the operating system level.

Our Novell licensing is up for renewal this Summer. We have been testing Ubuntu Server LTS and will make a decision later as to whether to abandon SLES.  To date, we find administration takes longer with Ubuntu (and Red Hat too) because there is nothing comparable to SUSE’s YaST (Yet Another System [management] Tool) in any other distro.  And once you “get” YaST’s peculiarities, it is incredibly efficient and keeps the entire OS in a highly consistent state.

In the interim, before you upgrade your SLES 11 system to SP2, talk to your application developers and make sure their application has been tested on SP2.

Take care,
Mark
CIO

Dropbox and Evernote are very easy to use and have enjoyed fairly broad market adoption.

Two issues we have with Dropbox are their security (all customer files were left open for several hours in Summer 2011 for anyone to see) and their Privacy Policy, which enables Dropbox to share your files with third-parties who provide support services to Dropbox.

Evernote’s Terms of Service have you granting Evernote a license to all of your Content that you post there. Similar to Dropbox, Evernote’s Privacy Policy also allows them to share your data with third parties. Worse, Evernote will drop cookies and tracking pixels on your devices.

For corporations in regulated industries (e.g. healthcare, financial services), employees who use such services for data covered by, say, HIPAA, may have created a defacto violation – Neither Dropbox nor Evernote to our knowledge execute Business Associate Agreements.

In unregulated industries, much corporate data is highly sensitive, so why would you want to allow a service provider to share it with third parties?

Solution: Zimbra Briefcase and CyberDuck

Zimbra already has a robust file-sharing, Google Docs-like offering in the form of the Briefcase. Until Zimbra releases Project Octupus in version 8, what is lacking in Zimbra now is the ability to synchronize easily the files in your Zimbra Briefcase with the files on your computer.

That functionality however is easily provided by a handy utility called Cyberduck, available for download at http://cyberduck.ch/. Historically, Cyberduck (and Filezilla, another favorite tool of ours) have been used for FTP transfers. As insecure plain-text FTP gave way to FTPS and SFTP, both Filezilla and Cyberduck expanded the number of transfer protocols supported.

But Cyberduck didn’t stop there. They saw that the future was in Cloud Storage, so they added even more secure transfer protocols to enable users to transfer files to Amazon S3 and indeed any storage repository which supports WebDAV over http — like Zimbra’s Briefcase.

So what we do ourselves and have configured for clients needing this functionality but are concerned about Dropbox’s past data breach history and Evernote’s content licensing, is to configure Cyberduck to talk directly to Zimbra’s Briefcase. Cyberduck you see, does Remote-Local Syncing of whole folders trees, so it’s a snap to keep your Zimbra Briefcase and your computer repositories in sync.

In the screenshot below, you can see in the upper right the Cyberduck window, looking at my Zimbra Briefcase.  In the upper left is the normal Zimbra web interface.  In the lower left is a local folder on my Mac, and in the lower right is the Cyberduck sync windoready to sync all of my Briefcase folders.

The sync process to be fair takes two mouse clicks; you have to remember to actually do it.  But if you need to keep all your corporate documents on your corporate Zimbra system and your corporate laptops, the combination of Zimbra and Cyberduck is a win-win until Zimbra’s Project Octopus comes along later this year.

Hope that helps,

Mark

Now, Zoho has several million users, including us, so fixing data corruption of that magnitude is not like letting Windows chkdsk just run for a few minutes after the server is rebooted. We’ll have to wait to see what the final outcome is, and for how long Zoho CRM (and SugarCRM and another 214 customers Equinix claims to host at that data center) remain down.

We suffered the same fate a few years ago at our former colocation host.  That and other issues caused us to move to a new colocation facility.  What happened at our former colocation host was that there was a power outage, the data center UPS (uninterruptible power supply) kicked in, and the system waited for the generator to start.  Only the generator didn’t start, and the UPS system had only a few minutes of juice in their batteries, so every server in the data center crashed, quite hard.  Fortunately, we had plans in place so we were able to recover quickly.

When we did a new colocation facility bakeoff, one of the detailed questions we asked was what happens if the power goes out and the generator fails to start?  Most data centers told us things like “We test the generator weekly! That won’t happen!” (which is what our former data center provider told us as well). Well, guess what?  You-know-what does happen periodically.

At the end of the day, we chose BayRing Communications, a New Hampshire-based phone company with two data centers at the old Pease Air Force base.  When we asked that same question of them, they laughed, literally, and said that in their experience gear fails all the time and so one needs to be prepared.  In their case, they bought a lot of batteries for their UPSs. When the power goes out, their UPS can run everything for several hours – plenty of time to either fix the generator or get a portable generator trucked in and hooked up.  They reminded us that, as a phone company, they get in big trouble if things like 911 don’t work for any length of time.

Indeed, at the end of the due diligence, we understood in more intimate detail what “carrier-grade” really means. And why, if you are running your own and hosting your clients’ mission-critical applications (like electronic health records, and email for regulated companies for example), “carrier-grade” has to be the minimum standard.

Does that cost more? More than some and less than others.

Will we survive without access to our CRM application through Zoho for a few hours? Sure. For a few days would be a real problem though.

At the end of the day, the takeaway here is that, whether you are taking care of a few dozen customers or a few million, when you choose a data center provider you really need to do your due diligence carefully.  Something clearly went horribly wrong at Equinix, and as of this writing, though power has been restored for a few hours, they haven’t disclosed the root cause.  We’ll have to wait and see…

If you have mission-critical applications and you have concerns about their hosting, we’d be happy to help you through a due diligence process that we organized for ourselves and our clients who host with us. Just give us a call at (207) 772-5678.

Mark

According to CSIS, “Up to 85 % of all virus infections occur as a result of drive-by attacks automated via commercial exploit kits.”

Commercial exploit kits are used both by the Bad Guys as well as by legitimate security shops to perform penetration testing, security scans and other tests appropriate for regulated companies and unregulated companies with valuable intellectual property to protect. Like guns, how these tools are used is up to the person with their finger on the trigger.

We have always known that patching your systems is very important.  Unfortunately, even though Windows Update now patches Office and other Microsoft software in addition to the Windows operating system itself, that’s not enough.  And now CSIS has the facts to prove it.

The five software packages that accounted for that 99.8% of all infections as reported by CSIS comprise:

1. Java JRE (which includes the browser plugin)
2. Adobe Reader
3. Adobe Acrobat
4. Adobe Flash and,
5. Microsoft Internet Explorer

While Internet Explorer gets patched via Windows Update, the other software packages have their own update system that prompts the user to update each software package individually.

Regrettably, in our experience way too many end-users ignore and click away those update warnings.  Further, in in more locked-down corporate environments, end-users often do not have sufficient rights to install software updates and patches; those updates and patches are pushed out to the end users’ machines via centralized system management software.  When the company’s system administrators do not push those patches out promptly enough, or the end users click away and defer updating those packages, a significant exposure is created.

And once one machine on a network is infected, many more are often subsequently infected.

So what does this all mean?

Well, first… this report reaffirms the importance of prompt patching.  Second, it documents that of the five top exposures, only one of them (Internet Explorer) is patched via Windows Update, so just turning on Windows Auto Update and thinking you are protected is at best fatuous.

At the end of the day, a comprehensive patching process is required; the proper execution of which is someone’s key responsibility.

If you have concerns about your company’s patching processes, please give us a call (207) 615-1529.

Mark

CIO

P.S.  You can read a summary of the study here.

The survey pointed out that workloads currently on clouds are comprised primarily of web sites, shared data storage,  backups and prototyping/sandbox.  Although most companies nowadays have adopted virtualization for line-of-business applications, the survey pointed out that very few of these of line-of-business have been migrated to the cloud.

At first, that struck me as odd since the greatest benefits of cloud computing come with cloud-ified (is that a word?) line-of-business applications.  And then the brick of realization hit me: Doh!  Most cloud environments aren’t ready to host line-of-business applications. Most line-of-business applications rely on high-performance databases (“fast disk I/O” in techno-speak), and most cloud providers have pretty slow disks — it’s how they keep prices down.

We ourselves spent eight months trying to get out of the hardware business, unsuccessfully.  Our Zimbra hosting farm hardware was approaching end-of-life, we had multiple clients who wanted to break the expensive cycle of premises-based hosting and so we had a raft of servers ready to be cloud hosted.

What we found was that the inexpensive cloud hosting providers’ infrastructure had nowhere-near-fast-enough disk I/O, and even very expensive private cloud hosting providers (at least the ones we talked to at the time) with very fast disk I/O had one or more “gotchas” that precluded us from going with them.

There are a lot of moving parts to successful cloud hosting. Not only do you have to be an expert at virtualization and eliminating single points of failure cost-effectively, but you have to be cognizant that most security standards are only a starting point. The most difficult challenge in moving clients to the cloud however is education/politics.  Cloud computing done well often requires business process improvements.

And then the second Brick ‘O Realization hit us (this was starting to hurt) that we have been doing virtualization for regulated companies, educating senior management and managing staff’s expectations for years.

So, we built our own private cloud, for our hosted Zimbra farm and for our clients. Architected explicitly for those database-intensive line-of-business applications requiring fast I/O. With 24 x 7 x 365 human client service. And priced appropriately, which is to say higher than Amazon (but not by as much as you think) and less than dedicated server hosting from the majors.

One client who runs SAS for healthcare analytics tested our private cloud and found it ran jobs anywhere from four to 10 times faster than their current blend of physical and virtual server infrastructure. Our own Zimbra hosting farm runs noticeably faster as well (and we do not overload our mailbox servers with the maximum amount of mailboxes either).

So in a nutshell, we think the big upcoming wave in Cloud Computing is migrating mission-critical, database-dependent line-of-business applications to private cloud service providers who “get” security, regulated environments and who can be relied on to manage, carefully and successfully, what can too often be a difficult transition to cloud computing.

And that’s why, after we had a chance to think about it, the survey from Cloud.com made sense and validated the investment we made in, and position of, our own private cloud environment.

If you are as scared of cloud computing as we were, give us a call at (207) 772-5678. You might rightfully decide that cloud computing is not for your company, but you might find out some things you didn’t expect.

Safe computing,

Mark, CIO

Reliable Networks Announces Private Cloud and Educational Series

Holiday Inn by the Bay – Wednesday, June 29^th 5:00pm – 6:00pm

PORTLAND, ME — June 13, 2011 — Reliable Networks, a network engineering and hosted services firm based in Portland, ME today announced its Private Cloud and Educational Series to help companies evaluate whether cloud hosting is at all appropriate for their business.

After nearly a decade of architecting and deploying private clouds for select clients, Reliable Networks has expanded the same successful formula to create the most secure and reliable cloud to hover over Maine.

Public versus Private Clouds

Outages and data breaches at large public cloud and service providers like Amazon EC2, Yahoo, Microsoft 360 and Gmail are reported with increasing frequency.  Niche private cloud providers like Reliable Networks however have been quietly providing secure, reliable services to clients for years.

Appropriateness of Cloud Hosting

Is the Cloud right for all businesses? Not necessarily. Businesses must evaluate whether cloud hosting is appropriate before even considering which type of cloud hosting is optimal, public, private, or hybrid. Reliable Networks has conducted a due diligence program to consider whether outsourcing this service would be in its clients’ best interests.  “Regrettably,” reports Reliable Networks President L. Mark Stone, “all of the private cloud providers we investigated had one or more areas of concern that made their offering inappropriate for our clients’ needs and for our own suite of hosted services.”

Cloud Hosting Educational Series

The temptation to move to the cloud is great, given the myraid benefits including cost savings, higher reliability and more efficient workflow processes.  “But as our own due diligence exposed, the road to Nirvana has a few IEDs buried along the shoulder.”  added Stone. “So we thought we should share the results of our cloud hosting due diligence with the community, to help others make better decisions about whether to, and if so, how to, host in the cloud.”

The first session in this Cloud Hosting Educational Series will be held on Wednesday, June 29^th from  5:00pm to 6:00pm at the Holiday Inn By The Bay in Portland.  Pre-registration is required (no at-event signups can be admitted) at www.reliablenetworks.com/events and complimentary refreshments, sushi, beer and wine will be served.

For more information:

Kristin Przybysz, (207) 772-5678, kristin@reliablenetworks.com

Lockheed and Northrop the articles further point out have already suffered intrusion attempts, with Northrop reportedly going so far as to shut down all remote access.

It’s not just defense contractors, Sony, VMware, Amazon, Google, and the State of Texas who suffer data breaches increasingly measured in the millions of records. We see typically half a dozen or so very professional intrusion attempts every day on our home firewalls; our data center firewalls see about the same.

SecureID, combined with a personal password known only to the user creates what is called a “two-factor authentication” authorization scheme.  Described as “something you have, plus something you know”, it works just like an ATM card (something you have) with your PIN (something you know). The two-factor authentication provided (past tense…) by SecureID often lulls users into a false sense of security and the temptation to use weak passwords; how many of us have 10-digit ATM card passcodes?  I used to have an 8-digit passcode but found it didn’t work on about half of all store credit card swipe pinpads.  Not terrific security…

And what happens when we lose our ATM card?  We cancel the card and get a new PIN.  Well… RSA just “cancelled” some 40 million SecureID cards.

With our without SecureID or some other two-factor authentication scheme, there is no substitute for good, basic password policies. We recommend strongly that our clients adopt password complexity, reuse and rotation policies, at least as follows:

1. Passwords should be a minimum of eight characters long and contain at least one each of:
   1. Uppercase character
   2. Lowercase character
   3. Number
   4. Punctuation mark or symbol (e.g. semi-colon, underscore, hyphen, parenthesis, etc.)
2. Passwords should be changed no less frequently than every 120 days (one company we know requires weekly password changes)
3. Passwords should not contain dictionary words or derivatives, or be based on personal information like birth dates or anniversaries.
4. Passwords once used should not be able to be reused for at least a year.
5. Lastly, the log files need to be parsed routinely for intrusion attempts (this can be automated) and a human alerted ASAP when something looks wonky.

Consider the costs to a company when a user’s email password is compromised and a hacker starts using that account to send out thousands and thousands of spam emails.  In short order, the company’s email server becomes blacklisted, no one in the company can send email anywhere, and business comes to a grinding halt.  It can take a few days to get off all the blacklists, so we advise clients to consider the costs of a few days of email downtime against the complaints from a vocal few users who don’t like changing their passwords three times a year.  It’s all about tradeoffs and risk management at the end of the day.

If you would like an objective review of your company’s security, remote access, password and related policies, give us a call at (207) 772-5678.

Take care!

Mark

The reasons for this coverage gap are several:  First, property and casualty insurance policies are written to cover tangible items and data isn’t tangible. Medical malpractice insurance policies don’t consider a data breach a medical error and so don’t usually cover the costs from data breaches. Even General Liability policies rarely include data breaches in the specific list of liabilities covered.

Worse, breaches are expensive! The estimated cost for a data breach spans $220 – $330 per record. Consider a primary care physician with a panel (i.e. patient base) of 4,000 patients. The practitioner’s cash out-of-pocket costs to remedy a typical data breach could exceed $1.0 million. Almost all states mandate some form of data breach reporting, and a quick search on DatalossDB.org shows health care providers are reporting data breaches frequently. Fail to report appropriately and your out-of-pocket costs go up; Stanford was recently fined $250,000 for failing to report a breach on a timely basis.

As businesses convert to electronic records or migrate to the cloud, increasingly more insurance companies offer cyber liability and data breach insurances. Rates vary depending on the risk within the practice. But it’s not easy obtaining cyber liability and data breach insurance. The vetting process is very thorough and time-consuming. We tell clients that the process is not unlike going through an actual security review — not a bad thing to do in any event since being “compliant” doesn’t necessarily mean you are “secure”!

The good news is that, with a combination of properly trained personnel and a secure network (not anywhere as expensive a proposition as you might think), any company can reduce the likelihood of these tragic and unexpected costs.

If you would like to see how your network security configurations compare with others, please feel free to call us at 207-772-5678.

Kristin Przybysz
Business Development

He should be: The Tenth Fleet has no ships, but is responsible for the cyber security (defensive) and cyber capabilities (offensive) of our Navy.

There were two key takeaways for me from this presentation.

We in the industry understand how easy it is for countries like Egypt to have “turned off” the wired Internet to their whole country so quickly, but it was interesting to see that the majority of the audience believes the Internet to be much more persistent and secure than it really is. The Internet is much more fragile than that. We as engineers know that Border Gateway Protocol (“BGP”) is the glue that connects the Internet together and controls routing of Internet traffic. Manipulation of BGP by malicious entities is often used for industrial and political espionage as one can, often with surprising ease, reroute selected Internet traffic through one’s own routers to analyze the entire flow of Internet traffic to a target. A good presentation on the fundamental security issues with BGP can be found on Renesys’s website here. (PDF Download.)

The second and more important takeaway for end users and our clients was the acknowledgment that malware these days is very professional (there is a very efficient global market for criminals and nation-states in malware and support services), well-hidden, and like StuxNet, very destructive and difficult to remediate.

The Admiral pointed out that good standard security practices, firewalls, security software and end-user training, though still very important, aren’t enough to defend a rich target. The Admiral cited Apple as an example of a company which has taken defense against industrial espionage seriously, and whose track record is yards better than most.

According to the Admiral, Apple uses software that collects data from server access logs, swipe card systems and a number of other systems and creates “profiles” of “normal” user activity. When a user’s activity deviates from normal, an alarm is triggered. This is a high-tech version of someone asking: “Why has Bobby been photocopying the new product design plans late each night?” and is similar to systems used in my former industry (investment banking) to track rogue traders and others potentially acting on inside information.

I can also tell you that we see on our data center and office firewalls at least a half-dozen professional intrusion attempts every day, most frequently from IP addresses registered in China, near Asia and Eastern Europe. Frankly, anyone who thinks that just because they are in Maine or outside of a large metro area that they are off the bad guys’ radar is deluding themselves; the Internet knows no boundaries.

If you have something valuable to protect and would like to benefit from our best practices, please do not hesitate to call us at (207) 772-5678.

Mark
CIO