I’m confused. I thought this blog is your newsletter, replacing your old newsletter site which is now an archive. And didn’t you mean you e-mailed not [snail] mailed the newsletter? And since you can’t re-mail your newsletter, is it now unavailable to me forever? Could you please explain?

OK, let’s see if I can clear up your confusion. The blog is the locus of the primary content that was previously Scot’s Newsletter (emailed and web page). And, yes, any newsletter I “mail” is emailed. But there’s a little more to this that you may have missed.

When the blog was launched, I merged the HTML and Text newsletters into a single list and began sending a notification newsletter to let people know whenever there is recently added content on the blog. For the most part, the notification message contains headline, blurb, and links to all (or most) of the blog entries since the last newsletter.

I expected to mail the newsletter on a roughly monthly schedule, but also promised only to mail the newsletter when there was something of benefit to the majority of subscribers. I polled people about this before I made the decision to switch to a blog and use the newsletter this way. Many people were not in favor of the blog, but the vast majority of existing newsletter subscribers wanted the newsletter to continue on as a notification message.

When I made the switch to the blog, I converted the newsletter to text only. Prior to that, the newsletter consisted of two separate lists, HTML and text only. We merged and de-duped the lists. De-duping means that when the same email address was found on both HTML and text lists, one instance was removed. We also cleaned the list so that recent bad addresses were deleted. The list size went from about 47K to about 43K.

Somewhat surprisingly, few people unsubscribed from the newsletter when I made the transition to a notification newsletter.

A key point

So, this may be the main point of your confusion, FuturePerfect:

Occasionally, I write exclusives for newsletter subscribers, which go out via email along with the list of links to the recent blog posts. Usually, these are not items of general interest to blog readers. Having been absent from subscriber’s inboxes for 5 months, though, I decided newsletter subscribers deserved some explanation. So, the email message contained a detailed explanation of why the newsletter hadn’t been mailed since March 27th.

Part of the reason for the unintended hiatus was a nearly three-month security battle I waged with hackers who penetrated the Web servers of my last webhost (IX Webhosting). As a result, part of the explanation included a quick overview of events and even some of my takeaways. I expect to publish a more detailed version of that story in the blog. I’m still not completely out of the woods (although it’s looking good), so I’ll hold off on my fully public report of my security challenges until the story is concluded.

So, if you’re someone who wants to stay very, very close to all things Scot’s Newsletter, subscribing to the newsletter is a good idea. It was my earlier judgment that most blog readers don’t need that much detail. It’s really your choice, however. If you’d like to subscribe to the newsletter, please use the Scot’s Newsletter Blog notification list sign-up page. It’s text only, is easy to unsubscribe from, and is likely to mail somewhere between 5 and 15 times a year.

Additionally, if you’d like to support Scot’s Newsletter, subscribing would be a fine way to do so. Although I have no plans to do this right now, it is at least possible that I might someday return to primary content being published in a newsletter format. More than likely, that would be after major new antispam technology became prevalent.

One of the reasons I moved to the blog was that the rise of Yahoo mail (the worst offender), gmail, Hotmail, and other free webmail services has meant the successful delivery rate of most newsletters — especially large HTML newsletters — has plummeted. Webmail services (I prefer gmail of the lot) tend to be quick to block newsletters. Because webmail is a free service, there’s no advantage to Webmail providers in delivering every message. There’s no service level they are compelled to meet. If they deem something to be in some way suspicious, then it’s blocked. And subscribers never even know.

A shorter, text-only newsletter has a much, much better likelihood of making it by the spam filters, though. So, for now, the notification newsletter makes a lot more sense.

Six reasons why the newsletter was on hiatus

So, because some people are interested, here’s a summary of the reasons I published in the newsletter explaining why it took a five-month breather. Unlike the newsletter, I’m placing these issues in chronological order. The delay to the newsletter — and the reduction in the number of my posts over the same period — was actually caused by a series of decisions and events:

1. I have young kids and a wife who is pursuing her own demanding career. My wife needs help, and my kids need time with dad. Nothing is more important than family, so over the last year I’ve been devoting more of my “free” time to my personal life.

2. Last fall I took on a new professional role as editor-in-chief of Computerworld. As you might imagine, the job demands a great deal of my attention. My professional responsibility is to the 39 editors, reporters, and writers of Computerworld’s editorial staff — even on my own time. Something has to give, and since Scot’s Newsletter is for the most part a moonlighting labor of love, it has been getting less of my time.

3. After delivering the Best Firewall Software of 2008 story in March — a decision and a story 18 months in the making — I gave myself a four-week break from even thinking about Scot’s Newsletter.

4. Over the last two years I have become a confirmed Macintosh user. Perhaps even more so after the latest detour into my own little security nightmare (see #6 below). Most SFNL readers are Windows users. I am still actively watching and using Windows, but I have recommended against Vista and there’s not a lot more to say about Windows XP. Until Windows 7 arrives, there’s a dearth of material to cover. The same thing happened at the end of the Windows 98SE/ME cycle, by the way. People take my lack of Windows coverage of late as a sign that I have abandoned Windows. While I have abandoned Vista, I have not abandoned Windows.

As long as I’m speaking of Windows, here’s some advice. Windows XP users: Please Install Service Pack 3. If you’re using an HP machine, make sure you have the latest updates (including firmware) for your computer before doing so. Also, Windows XP isn’t going to last forever. Windows users should be thinking about their next move within the Windows world. If you can afford it, have the hardware to support it, and you’re technically minded, Windows Server 2003 or 2008 might be an option. Windows Server is not for everyone, though.

Expect Windows 7 to arrive in 18 months or so (although Microsoft may not be able to deliver quite that fast). My best guess is that Windows 7 might wind up being something that might also be called Windows Vista Service Pack 3 with visible new changes (something like Windows XP SP2 or Windows ME). In other words, it may not be the fix that people who aren’t fond of Vista are waiting for.

In my opinion, Microsoft needs to downsize Windows. It has become bloated with non-essential extras. Simplicity, elegance, reliability, and performance should be the watch words of the next version of Windows. I wish I could say that’s likely to be the case, but I’m still hoping that I’m wrong.

5. When oil prices spiked in May and June, my interests turned sharply to a few of my other passions, automobiles, climate change, and alternative energy sources and technologies. Several posts were devoted to that, and I opted not to send a newsletter just for those posts, since most newsletter subscribers didn’t sign up for that type of content.

6. The Scot’s Newsletter sites were under security attack from May through early August. Once I discovered what was going on (in June), I was forced to shut down the blog and forums during this period, and I also spent well over 120 hours of my personal time fighting the problem on my home networks and machines as well as on my shared webhosting server. In the end, I had to research and select a new webhost, migrate my sites to the new host, update all my server applications, and resurrect the blog, forums, and other website in their new location.

The new webhost is much better than the old one in all aspects, including security, performance, and tech support. I’ve opted not to mention the new webhost by name (and I’d ask folks not to post the name if they decide to research it) until I write about it in a future post. What I can tell you now is that, so far, it’s the best webhost I’ve ever had. So there is a silver lining to my security hassles.

Not getting rid of me that easily

Finally, it should be noted that Scot’s Newsletter will continue to be updated as often as I have time to do so. It will continue to cover Windows, the Mac, broadband, security, Microsoft, Apple, and all the many things it has always covered. It may also cover other things I’m passionate about. Nothing has changed, really, it’s just that 2008 has so far been a year when I’ve had less time for writing posts. I’m not suddenly going to stop writing about computers. And I have no intention to discontinue the blog or the notification newsletter.

Because, these days, newsletters often accidentally wind up in spam traps, I wanted to make sure that newsletter subscribers got this little extra notification from me with a suggestion that they check to ensure they got their copies. (Please note: I cannot resend the newsletter to you if you didn’t get it. My apologies.)

If you’d like to subscribe to or unsubscribe from the Scot’s Newsletter Blog email-notification list, those tasks are quite easy. Please visit SNB subscription page.

– Scot

If you’re a frequenter of the forums, drop by and help us beta test the new forum software, configuration, and customizations.

Now that the forums are back open, the migration to the new host is complete. I’ve learned a lot from this less than happy experience. I’ll pass along some of my lessons learned in future blog entries.

During the hiatus, the blog was moved to a new webhost and was updated to the latest version of WordPress. Those jumps represent a major upgrade to the site, and I’m still working out some minor kinks. Please report any issues you may find by posting a comment to this thread or by sending me an email. Much obliged.

Welcome back. It feels good to be in business again.

– Scot

I have temporarily turned off the forums while I check its database for corruption. Perhaps that will improve performance here at the blog. We’ll see.

– Scot

Scot’s Newsletter does not accept any payment from any computer product vendor. I don’t do product consulting of any sort. I do not accept random payments of any sort from any computer software or hardware maker.

The newsletter has in past accepted newsletter ads (although not in many months). The only service an advertisement has ever earned from me is the advertisement itself. I have never accepted ads from any vendor whose products I was in the process of reviewing.

The only regular forms of revenue Scot’s Newsletter has are Google AdSense and personal contributions from readers. That funding does not cover my time and costs, nor have I ever attempted to make up that expense, just possibly defray some of it. Scot’s Newsletter is probably best described as a labor of love.

This is the point of this post:

My product recommendations have never been and will never be bought. Period.

Note about the elimination of newsletter ads:

The recent transformation of Scot’s Newsletter from a newsletter containing the content to a blog site with a notification newsletter changed the advertising proposition. A newsletter that merely links to content doesn’t support newsletter-based advertising very well. So eliminating newsletter ads was part of the plan from the start of the blog idea. I may investigate additional ad networks or programs like Google AdSense.

– Scot

It’s important to note that Scot’s Newsletter has become the Scot’s Newsletter Blog Notification List. It is now a text-only, blurb-and-link newsletter that links to the blog. I merge/purged the HTML and Text lists into one. There’s no need for those previously subscribed to the HTML list to unsubscribe and resubscribe. This change is automatic for all subscribers and completely invisible.

So, beyond that … where’s your Scot’s Newsletter Blog content? It’s coming. As I do every year around this time, I took a vacation the last 10 days of December — which pushed me back quite a bit.

Expect a software firewall update in the near future. I’m also testing NOD32 3.0 (so far so good for me, but I’m hearing about issues).

Ciao,

– Scot

But I was not expecting so many readers to be unable to read any of the stories at all, or to have to wait “minutes” for pages to load, during the first couple of hours after the newsletter mailed. Normally, the blog is very fast, snappy even. So the problems that many people saw were temporary. But I experienced the slow performance myself. Even though the problem lasted only about two hours until peak demand wound down, the experience for the people who respond right away to the newsletter is unacceptable.

In case you’re wondering why this happened, there are some underlying technical issues that I’ll come back to shortly, but the primary cause is that the change to a blog focus means that the newsletter no longer contains whole articles; it describes articles and links to them on the blog site. Everyone who reads a blog post by clicking a link in the newsletter is making a call to the database. Scot’s Newsletter has never worked this way in the past. So it’s a learning experience.

Since the last newsletter mailed on December 6, I’ve been working on solutions to the problem — and I’ve made some headway. Here’s a more detailed explanation of the technical issues and the solutions I’ve put in place since my last post on the topic:

Problem 1: Out of the box, the WordPress software has manual caching options but they’re turned off by default and not very easy to use. Without any form of caching, every single request to read a story requires a direct call to the database. I wasn’t aware that there was no default caching at all. I had read up on a WordPress add-on called WP-Cache that some people have had issues with. When I first installed WordPress, the available version of WP-Cache was older than the version of WordPress I installed. And the WP-Cache site does not list the WordPress versions it supports. I decided the better part of valor was to keep it simple at the beginning and not install too many WordPress plug-ins. The result, of course, was a WordPress installation that relied on database calls for every display of the content.

Problem 2: WordPress (in fact, most blog software) as well as Invision Power Board (and most forums software) require MySQL databases. For nearly five years, the Scot’s Newsletter Forums has had a large MySQL database. When I launched the blog I set up a second database for WordPress. What I didn’t know was that by setting up the database with the same user name that I use for the Forums, I was effectively forcing both databases to share the 50-concurrent-MySQL-database-calls limitation.

Fix 1: As it happens, WP-Cache was updated very shortly after I installed the blog. So after some more research on the built-in WordPress caching capabilities and WP-Cache, I opted to go with the plug-in. It installed cleanly and easily (although, it could use more detailed installation instructions). I experienced none of the problems that some people reported after installation. And I can see from its control panel that it quickly indexed and cached the stories.

Fix 2: I also wanted to separate the connection limit on the forums database from the connection limit on the blog database. My thought was: Could I get a second account from my webhost and move one or the other to that account? Doing that would require me to change the URL of either the blog or the forums — for very good reasons on both sides, I didn’t want to make either change if I could avoid it.

It took several days of back and forth with IX Webhosting, my webhost, before the tech reps finally understood my problem and one specific rep (Alexander, who has been a great help in past) finally understood all the aspects of what I was asking for and gave me a useful answer. The answer when it came was worth the several days of low-grade frustration that IX’s less-experienced tech-support reps put me through. All I had to do was create a new user name for the blog database, eliminate the access to that database from the shared user name originally created for the forums database — and each database would have separate 50-concurrent-users database-connection limit. There was no added charge for this, I didn’t need a separate account, and most importantly, I would not be forced to make a very tough URL-change decision. It was also an easy thing to accomplish — less than 10 minutes after I read Alexander’s post in my trouble ticket, I had made the suggested change.

My hope is that, with an effective increase in the upper limit on concurrent database connections (since blog and forums no longer share) and the addition of the performance-improving, database-call-reducing cache software, the blog will be sufficiently better able to serve stories that it will hold up under the steep initial demand for stories during the first few hours after the newsletter mails.

It all sounds good, and the blog seems snappier to me. But we’re also going to need the proof before I dust off my hands. I’m currently thinking that the next newsletter will mail sometime the first full week of January, so I’ll be monitoring the situation next time to see whether the problems have been solved or whether additional measures are called for.

Notification List Subscribers: Let me know if you run into performance issues when trying to reach the blog during the first few hours after the next newsletter mails.

So, to summarize … Scot stupid, Mike Nash good Samaritan, blog site just fine.

Many of you wrote to me to tell me you either got error messages or had to wait minutes for individual blog stories to load in your browsers. Some of you just assumed that this was the normal experience at Scot’s Blog, and that is just not the case. Most of the time it’s pretty snappy.

The problem boils down to this: WordPress (and all other blog packages I’ve inspected) use a database to store the stories in. Many of them, including WordPress, use MySQL. My forums also uses a separate MySQL database. The two databases are on the same server. And the arrival of the blog newsletter quickly maxed out the number of simultaneous connections that my webhost has limited my MySQL service to. So people either got “database connection” errors or just waited and waited for the server’s time to free up. The forums also experienced interruptions and slow page loads.

An hour or two later, the problem had disappeared as the initial rush for the newsletter slowed down to a steady stream.

Solutions

Clearly, I have to separate the forums database from the blog database. Those two can’t co-exist on the same server with a shared limitation on concurrent connections. The only way I can do that, however, is to change the domain name of either the forums or the blog.

Then I have two choices: Either I purchase a new, separate account at the existing webhost or I move either the blog or the forums to a new webhost.

I just wanted to announce to all who experienced problems yesterday that I am actively working on this issue as a top priority. I can’t promise it will be fixed by the next issue of the newsletter — but if not, it will be explained in the next issue of the newsletter. And this will be resolved.

Update on December 8: Please see this addendum in the form of a comment to this blog post. I’ve added caching software to WordPress, which may solve or at least alleviate some of the problem.

Update on December 15: The MySQL databases for the Scot’s Newsletter Forums and Scot’s Newsletter Blog have been reconfigured so that they will have separate maximum concurrent connections. The change was easy to configure, once my Webhost explained it to me, and it didn’t cost anything.

With these two changes — the cache and separation of concurrent connections — I think we’re ready to give this another go with the next newsletter. If we continue to have performance and time-out problems during the two hours following the initial sending of the newsletter, I will look for another webhost — a solution that won’t happen as quickly as these changes did. But what’s the point of having newsletter notification if the server can’t hold up to the demand? That isn’t acceptable.

There, I got that off my chest. But it’s not what I wanted to tell you.

Now, about the blog, newsletter, whatever it is — Scot’s Newsletter. Here’s where things stand.

There’s a lot to do on the blog and newsletter sites to stitch them together, make them work in tandem. I want to combine them, not replace the old with the new. I’m feeling my way toward that, gradually. But at some point, major surgery will probably occur. For now, my recent promotion and ongoing physical therapy take priority. But I am planning in my minimal spare time.

Performing surgery on the WordPress code — except where related to design — is proving to be more difficult than I’d hoped. And the mods available for WordPress aren’t all that great. I’ve received lots of excellent suggestions from the newsletter’s readers for making the blog work better. People generally would prefer that the blog entries weren’t separate permalink pages. Many would also like the same kind of navigation the HTML newsletter offered, with a TOC at the top linking down to each item and a “back to top” link at the end of each entry. I’d prefer that too, but it’s not as easy to refit the blog code as you’d think.

Another recurring theme I hear from readers is the desire to make it easy to print “an issue.” The concept of separate monthly issues is blurring, to be honest. Much quicker than I’d expected, I find myself writing individual blog posts on different days. It seems to me that the content is better when I write it this way. But it’s a judgment call as to what items are in a specific issue. The blog software archives by calendar month.

So, anyway, the problem with printing isn’t the printing itself, but the fact that you’re not printing an issue, per se. Also, you need to print the stories pretty soon after you get the text-based blurb-and-link newsletter because as I add new posts, ones from the bottom roll off the main page into the archive.

What I’m currently doing is leaving all the latest entries at their full length, and the ones from the “last issue” will be abbreviated with a “read more” link. I’m also setting the “Recent Posts” column to show only the stories in the current “issue.” I don’t know where this will lead, to be honest. This could be transitional, or it might be my solution. I still have a lot of thinking to do to make the user experience the way I want it.

One thing I want to make very clear: This Is Much Easier for Me! I appreciate the support of so many of you who agreed to something very different even though you might not really have wanted that change. Everything about this is better, easier, faster, and less expensive. It also lets me focus researching and writing content instead of the tedium of production hassles.

The New Newsletter

Those of you getting the notification newsletter are seeing the first rendition of a new template this week. Drop me a note if you have a suggestion. It’s a work in progress.

All newsletter subscribers should be advised that my newsletter distributor, Dundee.net, will shortly be merging the HTML and Text lists into a single list for the notification newsletter. That change saves me money and time, but the main reason to do it is that it will greatly simplify the subscription tools. Subscribing, unsubscribing, and changing your email address all become much more straightforward processes. You’ll still need to receive a confirmation email and click the confirmation link it contains to verify intent to subscribe (also on email-address changes), but most of the problems that people have with the Scot’s Newsletter subscription tools will vaporize when there’s only one list.

What does merging the two lists mean for you? For most people, nothing. Both the process and the result of the merging should be invisible. People who are currently subscribed to both the HTML and Text lists with the same email address should find that they’re subscribed only once. If you’re subscribed to both lists with two different email addresses, you’ll have two subscriptions to the same newsletter after the merging. You can simply unsubscribe one of them.

During the merge period, the subscription center may be down for hours or days.

Well, that’s it for now. I’ll keep you posted on changes as I schedule them.

I recently underwent joint-replacement surgery and was in the hospital for a few days. I’m still in the process of several weeks of recuperation and physical therapy, but I am starting to feel human again.

I’ve got some additional stuff in the works. So please check back.

In the text-only, October 2 issue of the newsletter, I described two possible format changes for Scot’s Newsletter:

Option 1. A blog presentation with a text-only, blurb-and-link notification newsletter.

Option 2. An HTML-only newsletter, without a text alternative.

The main reason to change formats is to save production time for yours truly. It takes almost as much time to produce and send the newsletter as it does to write it. (Of course, the majority of the time I put into the newsletter is spent on research.)

Now five days after the newsletter went out, I’ve received over 2,700 votes. With voting beginning to slow down, the votes for the blog outnumber the HTML-only newsletter option 2:1. In other words, I’ve received a little more than twice as many votes for the blog as I have for the HTML newsletter.

There is much to be done on the blog site, in revising the template for the new text newsletter, and on the Scot’s Newsletter website to pull this altogether. It’s not all going to happen overnight.

Several people who voted requested navigation changes to the blog site that I’d already been thinking about. I’m also considering how to work through the archiving system, how to make the site print better (although it does print OK for now). Please be patient while I make modifications, which will come over time — a little bit here and a little bit there.

WordPress geeks and designers: If you’ve got experience, I might need your help!

For those of you who wanted HTML or who preferred the text newsletter, I ask you to give this a good try. And let me know what isn’t working for you. I can’t promise to fix all problems, but I do care and will fix anything I can.

Thanks to all those people who voted, and the many people who expressed their intention to support Scot’s Newsletter no matter what change I went with. That means a lot to me.