A (refactored) Christmas Carol

Our story begins on a cold and bleak desktop. Excel, an aged application , ignores an invitation to Christmas dinner from his nephew UserVoice. Excel turns away desperate pleas from 222 voters who seek an upgrade to Tables in order to provide data integrity and usability for their poor users. Excel only grudgingly allows his overworked, underpaid Tables to be used while a sheet is protected, and even then, only to conform to the bare minimum of the existing Sheet Protection functionality.

At home that night, Excel is visited by Lotus 1-2-3’s ghost, who wanders the Earth, entwined by heavy patents and dependency chains forged during a lifetime of greed and selfishness. Lotus 1-2-3 tells Excel that it has one chance to avoid the same fate: Excel will be visited by three spirits and he must listen to them or be cursed to carry dependency chains of his own, much longer than Lotus 1-2-3’s chains.

The first of the spirits, The Ghost of Microsoft Past, takes Excel to scenes of Excel’s boyhood and youth, reminding him of a time when he was more innocent. The boyhood scenes portray Excel’s lonely childhood, his relationship with his beloved sister Word, and an office party hosted by his first employer, Mr. Gates, who treated Excel like a son. They also portray Excel’s neglected fiancée Access, who ends their relationship after she realizes that Excel will never love her as much as he loves unstructured data.

The second spirit, the Ghost of Microsoft Present, takes Excel to a joy-filled help forum of people gathering the makings of a PowerQuery, and then on to a celebration of business intelligence in a data-miner’s PowerPivot data model. They then visit Acme Inc, where we meet Jolly Jeff, an otherwise happy analyst who’s Table-driven macro-free Excel templates are seriously unprotected. The spirit informs Excel that Jolly Jeff’s carefully constructed templates will soon be corrupted by ignorance and laziness unless the course of events changes. Before disappearing, the spirit shows Excel two hideous, emaciated columns of data named Calculated_Column and Freetext. He tells Excel to beware the former above all and mocks Excel’s complete lack of concern for their unprotected state.

The third spirit, the Ghost of Microsoft Yet to Come, shows Excel a business office in the future. The ghost shows him scenes involving the death of a disliked application. The application’s funeral will only be attended by local businessmen if donuts (of the charting variety) are provided. When Excel asks the ghost to show anyone who feels any emotion over the application’s death, the ghost can only show him the pleasure of Stephen Few. The ghost then shows Excel the application’s neglected grave:

Excel: 1985 – 365.
Here lies one who frivolously cavorted with the entire office while not using adequate protection.
Recalculate AND Die.

Sobbing, Excel pledges to the ghost that he will change his Table Protection to avoid this outcome. Excel awakens the next day a changed program. Excel spends the day with Mr Gates’ family and anonymously sends a large update to Tables to the Weir home for Christmas dinner. From then on Excel began to treat everyone who uses Tables in a protected workbook with kindness, generosity and compassion, embodying the spirit of User Experience.

What I want for Christmas

…is for Microsoft to fix how Sheet Protection and Tables interact. I want it more than world peace. I even want it more than cold beer. See…I even put it on my Christmas wish list:

What am I talking about? That template above is a great example: Let’s say I want to send you all some templates for you to fill out and send back. Being a prudent developer, I’ll use Excel Tables, and those tables will have a mixture of Calculated Columns with the formulas locked down (because I don’t trust you one bit), and free-text fields (because I want you to think that I care about what you have to say).

That’s what I’ve done above: I’ve got a RANK column that automatically assigns an incrementing integer to each entry, and I’ve got a freetext field where you can put your own requests, if for some reason you don’t share my world view. And I also want to let you insert additional columns, in case you want more than three things.

I better push Ctrl + F1 to bring up the Format Cell dialog, and make sure those genius formulas in that Calculated Column will be protected when the sheet is locked:

…and I better make sure that the ‘Request to Santa’ column is NOT protected:

Awesome…all I need to do in order to activate this protection is to lock the sheet, while still allowing row inserts and deletes:

And all is good, until some moron decides that Cold Beer comes in a distant third behind some additional thing they would like to add to this Christmas wish-list, like “New President”. So they right-click within the Table Row while Tra-la-laa-ing, only to be stopped mid laa: the Insert > Table Rows Above option is greyed out. :-(

But being persistent, they decide to force the issue by selecting the entire row, and then right clicking. And indeed, this time they encouragingly see that indeed the Insert New President… option is still available to them:

…only to find that Excel has trumped their desire:

But weirdly, when they click OK, Excel decides that it can in fact do it, kinda:

It begrudgingly inserted the new row, but it didn’t autofill the formula in the protected column. Meaning while our hapless agitator can fill out the freetext field with their new second-greatest wish, they can’t rank it. Meaning Santa might simply ignore it as invalid input:

God forbid they would want to do something equally trivial, such as extend the Table by either trying to type underneath it, or by selecting the bottom right cell and pushing Tab or Enter, because that accomplishes nada. As does Sorting and a myriad of other things that the Protect Sheet dialog swears it will let you do.

I just cannot fathom why this bug hasn’t been fixed. I recall Zack Barrasse talking about it years ago, along with some other stuff he’s mentioned on the Excel UserVoice site. Sure, I could write some VBA to add a custom ‘Insert Row’ entry to the right click menu, but I try to keep my templates macro-free as much as possible. Some absolutely need to be macro free. And I wouldn’t need macros anyhow if this functionality actually did what it says right there on the box.

Please go vote for this. And Microsoft, please go fix it…It’s got 191 votes already, and counting. And it is just sooo broken.

BBj Developer Wanted

I’m looking to hire a BBj developer in the Omaha, NE area. Why would a reader of an Excel blog care? I’m glad you asked.

BBj used to be called BBx. And that was formerly known as Business Basic. If you’re a decent VBA or VB6.0 programmer, BBj will look pretty familiar to you. I’ve been coding in it for that last couple of months and, other than the constant trips to the documentation for syntax help, it’s going pretty well. The official IDE is Eclipse with a plugin, but we’ll be switching to UltraEdit very soon.

Here’s the basics

What you’ll do:

  • Fix bugs reported by users
  • Add new features, both large and small, as business needs dictate
  • Enhance existing features for stability and usability
  • Monitor system health and correct issues timely

Required Skills:

  • Bachelors degree and three years of software development experience or
  • Equivalent software development experience

Desired Skills:

  • Experience programming in BBj, BBx, or Business Basic
  • Experience programming in Visual Basic 6 or VBA
  • Experience in the travel center, c-store, or wholesale fuel industries
  • Experience with Subversion, UltraEdit, and iTop

Send me an email at dick@kusleika.com if you’d like more information.

International Keyboard Shortcut Day 2017

Another November. Another first Wednesday. Another International Keyboard Shortcut Day. The day when people from all over the world become far less efficient in an effort to be more efficient the rest of the year.

Let’s mix it up a bit this year. Instead of me listing various levels of participation, I want to turn you into an evangelist. No, you won’t be required to best the devil in a fiddle playing contest or anything like that.

Today, tell someone else about a keyboard shortcut you like. You can, for example, casually mention to a co-worker how much you enjoy using hyperlinks since you learned the Ctrl+K shortcut.

Even better, you could exclaim loudly throughout the office how you wish there was an easier way to switch worksheets in Excel. Someone may yell back “Just use Ctrl+PgUp and Ctrl+PgDn” thereby educating the whole office. If nobody yells back, find a willing confederate and give him the answer and instructions about how to yell back.

Dramatic reenactments are another effective method of communication. Stage a skit in the cafeteria about an office worker at her wit’s end. You see, she has such a long list of sub-folders under her Inbox and the one she wants to click is never in view. She always has to scroll. Then she learns about Ctrl+Y and, later that day, becomes the CEO.

You might hear things like “Get out of my office!” or “Stop shouting. We’re trying to work here!”. Don’t be discouraged. Our message must be heard.

Renumbering Arrays in Code

I’ve got this bit of code where I’m listing table fields that I’m going to eventually Join into a SELECT statement.

As you can see, I needed to add a new field in position 1. Now I’m faced with renumbering the rest of the array. Terrible. So I wrote this:

Now I can copy the code, run this procedure, and paste the results.

Ahhh. Satisfying. Here’s how the stuff inside the loop works.

This splits the line into:

vaLine
0 fields(17
1 = “BOLState”

This results in:

vaLineStart
0 fields
1 17

Then I just concatenate the relevant parts back together with a different number.

Counting Files by Date

Someone told me we are posting more frequently lately. (For non-accountants, posting means taking the entered transactions and updating other files with the information.) Ever the skeptic, I decide to see for myself. Whenever we post, we produce a pre-post report in the form of

Pre-Post_Sales_Journal_yyyymmddhhmmss.TXT

PATH is a module level constant pointing to the folder.

If this was more than a one-off program, I would have written this line in a way that you could read it. The inner Split creates an array like

[0] = Pre-Post_Sales_Journal_yyyymmddhhmmss, [1] = .TXT

and I take the first element (the zeroth index) of the array. Then I split that further

[0] = Pre-Post, [1] = Sales, [2] = Journal, [3] = yyyymmddhhmmss

and I take the fourth element (index = 3) of that array. That’s my date in string form.

I put a bunch of dates in column A of sheet1 for as far back as I wanted to go. Then I add 1 to the cell to the right of the date. It turns out we are posting more frequently.

Inconsistent ListRow Copy

Here’s a cautionary tale for you. Let’s say you want to let users copy rows from a Source table row by row to a Dest table, by pushing a button:

…so you whip up a bit of code like this, and assign it to that button:

And when you run it, it works just fine:

…until that is, someone hides an entire column in the source table, sets a filter that similarly hides some rows, and leaves a cell in that Table selected before running the code again, in which case you get this:

As you can see from the above:

  • For any ListRows in your source that happen to be visible, only the cells from the visible columns get copied, to a contiguous block in the Dest table, but
  • For hidden ListRows, all cells get copied

Add to this the fact that everything works just fine if the user happened to select a cell outside the table before triggering the code:

…and you’ve got the makings of a hard-to-diagnose bug that will eat up hours of your time trying to replicate.

The fix? Don’t use the .copy method. Just set the values of the second range directly to the values of the first:

…which works fine, and is faster anyhow:

In case you’re wondering what happens if you bring the whole DataBodyRange through from the Source Table using that dangerous .Copy method i.e using code like this:

…then again the results depend on whether a cell is selected in the Source table:

…or not:

Again, avoid the inconsistency by setting the values of the second range directly to the values of the first:

…which works fine, fast:

Here’s a sample file:LRCopy Test

Always Use Stored Procedures

I take data that has been entered in Excel and I store it in SQL Server. A lot. I do that a lot. The proper way to do that is to create a stored procedure for every database operation you need and to execute that stored procedure from VBA. The quick and dirty way is to build a SQL string and execute it. As you might have guessed from the title, I chose the quick and dirty way and was recently bit in the ass.

Here’s the long and the short of it: Some numbers got formatted as dates and it really screwed stuff up. I had some code that looked similar to

The field ManifestID is a BIGINT and vaData(i,1) contained 4/15/2023. The ManifestID was 45031, someone (me) mistook that for a date that lost its formatting and promptly fixed (broke) the formatting. I noticed that several dozen entries in Blend had a ManifestID of zero. SQL Server dutifully took 4/15/2023, did the division (4 divided by 15 divided by 2,023), came up with zero, and put zero in the field.

After some self-flagellation, I wondered if a stored procedure would have caught this error. I assumed that when I tried to pass a date into a BIGINT parameter, the code would error out and I would have avoided this whole mess. But I was wrong. Instead, the stored procedure converted the date to its integer value – not by dividing like in the SQL String method, but by some conversion that I didn’t think was possible. Excel stores dates as the number of days since 12/31/1899. That’s not unique, but I’m pretty sure SQL server doesn’t store them that way. And how would ADO or T-SQL know to convert it in that way?

I devised a test. First create a table

Next, create a stored procedure to insert records

Then I wrote some code to insert rows

In the code, I define two formats in an array: General and m/d/yyyy. I loop through that array and apply the formats to cell G1 where I have an unsuspecting integer. In the first pass, it’s formatted as General and looks like a proper integer. I build up a INSERT INTO Sql string and execute it right off the connection. Then, still inside the loop, I do it the right way: Create a command object, add a parameter, and execute it.

In the second iteration of the loop, cell G1 gets formatted as a date and it all happens again.

I was expecting an error, so I had an error handler that printed out the whole table whenever thing bombed. But it never bombed. It executed just fine.

With the integer formatted as a number, both the string method and the stored procedure method inserted properly. That’s the first two 45000’s. The third 45000 is the string method when the integer is formatted as a date. That’s the one where SQL does division. The last 45000 is the one I thought would error out. But passing in a date to a BIGINT parameter converted it to the proper number. I even put G1 into a variant array to simulate my real world situation.

I still don’t know, and am interested to know, what is doing the conversion. But in the meantime I’m happy to learn my lesson and vow to use stored procedures like a good boy.