Bubbles are supposed to happen only once in a lifetime, but we have had at least three in the last decade! Baby boomers and everyone else look out!

 

The last decade has brought the dotcom bubble, the housing bubble, and now the Baby Boomer Bubble. Never heard of the Baby Boomer Bubble? Just talk to anyone nearing retirement, and you may quickly find out that there is a hidden bubble that is now bursting. Baby boomers have been investing large amounts of money for most of their lives, much more than any previous generation. Now they are losing tremendous amounts of money and are slowing becoming very angry.

Boomers have been taught to be frugal and save money while the rest of the society been on a credit induced spending orgy. Unfortunately, our society’s receptiveness to debt has now infected the Boomers savings, which may have been cut in half during 2000-2003 and now cut in half again during the great unwinding of 2008. This would be bad enough, except that 2008 is not over. If this keeps up, Boomers may start to invest in under bed mattress storage bins for the storage of their cash.

The Baby Boomer Bubble is not a single event, but an aggregate bubble across the stock, real estate, and bond markets. Basically wherever the Boomers have put their money has gone bust! Even reverse mortgages will probably not work well with the drastic drop in home prices. So what investment is left that is safe for Boomers? Are the only things left cash and T-bills? Are the Boomers angry enough to cash in their chips?

Not a Boomer so you don’t care? Well, consider what the next couple of years holds – the largest most politically active group in the US (the Boomers) is angry and demanding action, the finance system is tanking, the federal government now owns pieces of banks and soon insurance companies, the President will likely be a Democrat in two weeks, the House of Representatives will definitely be run by the Democrats, and the Senate may have a filibuster proof 60 Democrats. Want change you can believe in? How about change that you are powerless to stop?

This is not to say that everything that is coming is bad, there is widespread agreement that the healthcare insurance system is broken and that a realistic long term energy policy is needed. It is clear that Republican “leadership” in these areas was either non-existent or being used as political bargaining chips. The energy policy of the Republicans seemed to be “drill baby, drill”, unfortunately we got drilled. The only thing that looked like a Republican healthcare insurance plan was to reduce taxes on cigarettes.

So, where does this leave us? Two weeks out from a US election that is likely an inflection point in history. Not only does it impact the US, but the entire world will be impacted. The US military is fully engaged in two wars, the financial system is in turmoil, the economy is slowing with unemployment likely to go double digit next year, and individual debt is exploding. Will a change in the political situation in Washington fix all of this?

Enron, Securitization, and the 2008 Financial Collapse

 

The famous saying is "those that do not learn from history are doomed to repeat it." Unfortunately the core mistake made by Enron was learned for the purpose of repeating it with the entire market! The key mistake Enron made was the devaluation of physical assets and the overvaluation of financial (i.e. phony) assets. Enron’s core business was the operation of a natural gas pipeline until they decided to become the master of the energy securitization market, which lead to them becoming an energy broker.

The operation of a natural gas pipeline required large amounts of capital and staff devoted to physical tasks and did not allow for the rapid (albeit artificial) growth that the executives of Enron desired. Instead of figuring out how to grow the pipeline or better manage the costs, Enron decided to switch its focus to becoming the world first brokerage house for energy. Enron then diversified into electricity and water utilities going away from their core natural gas competencies for the purpose of gaining a competitive knowledge base outside of natural gas. Enron even tried to get in on the new “Broadband Internet” craze considering internet bandwidth as a tradable commodity. This switch allowed Enron to free itself from financial realities inherent to its business and act like a financial institution.

The biggest concept that the market learned from the Enron collapse is that the role of the accounting department can be switched from oversight and control to revenue generation! This revenue is not created by effective accounting practices that lead to tax savings or from deep audits that root out corporate corruption but are produced by repackaging of financial assets in such a way that their value is not immediately apparent. This repackaging allows for an intentional or accidental (you decide which) overvaluation of the assets, thus revenue is generated.

Fortunately for those of us that did not invest in Enron, we were not directly impacted by their actions. Enron packaged up bad assets and sold it back to itself structuring it is such a way as to artificially increase its book value (while in fact acting as an anchor around Enron’s neck tied to its stock price). This was done to feed the hoards of crazed investors that were willing to pump up Enron’s stock price if it “met its numbers”, without regard to the fact that Enron was cash broke. If only this were the case for the market debacle now.

Hillary’s only road to the White House requires McCain to win in 2008

Microsoft has a distinct advantage over its competitors because it uses its own tools to create its products.  Historically, its only perceived meaningful competitor was Borland, which was why Microsoft was so joyous when Borland became irrelevant.  Even so, Microsoft has been able to use Office and Windows as smoke screens for its actual intentions – domination of the internet.  Described here is how I perceive Microsoft intends to accomplish this.

The public concept of the Internet changed in 1995 from being an interconnection of networks and associated tools to being viewed solely as the World Wide Web.  News groups, gopher sites, and telnet bulletin board systems were the main applications of the internet prior to that time.  The web was introduced almost as a logical replacement of the menu based gopher servers with hyperlinks instead of the gopher menus for navigation.  It is no wonder that Microsoft was not the first on the web bandwagon, but the introduction of WPF and XAML cleary signifies a change.  If Microsoft is successful in implementing its strategy, the public’s perception of the Internet will change from a data dominant format (web pages) to an application based format (WPF applications).

WPF and XAML are not about making web pages more interactive or flashy but are intended to obsolete web pages.  Instead of trillions of documents linked together, the web would be replaced by seas of Internet applications.  These applications will use web services for interaction instead of HTML hyperlinks, eliminating the need for web search engines.  The internet will slowly become opaque to the search engine crawlers because interpreting XAML application code will not yield the same sort of meaningful results generated from HTML documents.

The introduction of WPF will generate a market for off the shelf internet applications that never fully existed with HTML.  The imposed connectionless nature of HTTP that hampers HTML is removed along with gaining the abilities to easily use local processing and storage. This can be leveraged to remove the significant privacy concerns that many users have related to web sites while significantly reducing the need for web server capacity.  WPF enables the creation of new forms of internet applications.  For example, instead of newspapers maintaining their own separate websites, a single browser based internet application could be created that would be able to access any of their content (either fee or ad based).  But, how would internet applications be found if they are not searchable through Google?  

Some of my more embarrassing learning experiences of being a father for the first time are things that I would ordinarily prefer not to share with others.  For example, the first time I changed a diaper is something that I would rather keep to myself and would have except for the fact that it went so poorly.  The worst part about it is that I knew how to change a diaper, but sometimes circumstances override one’s reason, which is how the great diaper mishap of 2002 happened.

It all began innocently enough.  The night after the birth of our first child, we were in the hospital and everything was going well.  Nurses would come in periodically to help us with our baby and had showed us how to change our baby’s diaper using gauze pads instead of baby wipes.  The nurses explained that baby wipes could be too harsh for a newborn and that we should use gauze pads for the first week instead, which seemed reasonable.  The only problem is we did not see the nurse wetting the gauze before using it, but how could that matter?

After settling in for some sleep on the fold-out couch provided for fathers in the hospital recovery room, I knew that the chance of sleeping more than a couple of hours was slim.  Even so, any sleep was welcome.  Waking to the sound of our newborn crying, it was about 4AM and we knew immediately that it was time to change her diaper.  Should we ring the nurse to help us? Of course not!  A diaper change is not that big of a deal and what could go wrong?  Undoing the diaper, it was obvious that our parental instincts were right on target.

The poop was a dark color, similar to a spinach salad that was put in the blender.  So I began to wipe with the gauze pads, but immediately noticed something was amiss.  Instead of wiping the poop off, it just smeared it around and made everything worse!  I remember my wife’s words which were a half laughing “oh no!”  Realizing that I am a man who just made a terrible blunder, I did the only thing that I could do as a man, which was to get more gauze and continue wiping making the situation even worse.

The rise of the Microsoft as the child of IBM

There is one dominant company that impacts computing.  Even if you do not use their products, you are impacted by their decisions.  They are a one stop shop for operating systems, databases, and software development.  Is there a particular company that comes to mind?  Well, twenty to thirty years ago, it would have been IBM.  Now that company is Microsoft.

Before I go any further, I should explain one important point.  Do not let this point mistakenly cloud the bigger idea that comes later.  I have spent the last decade or more making money using Microsoft products.  Prior to that, I was a kid and beholden to no company, especially IBM.  Even so, I was a geeky kid and very much integrated into the computing community, which included products from Atari, Commodore, IBM, and Apple.  I experienced the thrill of 8-bit, 1-Mhz, 48 KB Atari computing and thought that IBM was the big nasty behemoth that was forcing everyone to suffer with choosing between four colors and no colors.

Even so, I was excited the day my father brought home an IBM-PC, if for no other reason than it was something new to learn.  I became an expert at switching diskettes from one drive to another, a skill that was necessary to properly execute Lotus 1-2-3 successfully.  I also experienced the whir of a 10MB hard drive, which was equivalent in my thinking to a 100 TB drive today in size.  The only downside was that we almost got a Macintosh instead, which probably would have transformed me from a geek into a geek whose father spent a large amount of money on being a geek.  So I got to see the Atari/Commodore/Apple battle along side the IBM/Clone wars that started to occur in the mid to late 1980s.

The common thinking, as far as I knew it, early on was that MS-DOS was only an interim OS until CPM or some UNIX variant matured enough to take over.  Of course, I was not in on the meetings between IBM and Bill Gates where he sold them something he did not have, so I could not have witnessed the rise of Microsoft from the ground floor.  But, I did witness the rise of Microsoft Windows, mainly because I was interested in an alternate product called GEM (Graphic Environment Manager, or something like that) which was a solid competitor to early versions of Windows and was included with new Atari ST Computer (welcome to the 16-bit world!).  I also witnessed the rise of Microsoft Office, which took multiple iterations until it crushed the entrenched competitors like WordPerfect, Paradox, Lotus 1-2-3, etc.  But the biggest morale boosting win for Microsoft did not come from the operating system market or office suite market, but instead it was on the development tools side.

Microsoft’s core concept of itself is as a software development tool company.  This permeates everything Microsoft develops, even Windows and Office.  Linux definitely threatens or threatened Microsoft’s dominance in the operating system arena.  There have been multiple attempts (some feeble) to dislodge Microsoft from their dominance in the office suite arena.  But nothing, not even the excellent free development tools available (Eclipse being one) have caused Microsoft to change their course, although it possibly led to the free express versions of Visual Studio.  There is one competitor, whom Microsoft lovingly called “Buck Forland”.  If you switch the “B” and “F”, you will understand.

The earliest IDE (Integrated development environment) that I can remember is Borland’s Turbo C.  You could write, compile, and test the code all within a single application, which was a huge advance in development tools.  This innovation may have led Borland to make an excellent development tool for Windows called Delphi.  If Delphi was a Microsoft product, it would have been the dominant development tool and Visual Basic may not have survived.  I say this even though for most of my career I would likely have been called a Visual Basic developer.

Many late to the game may look at Microsoft first through the lens of the Netscape/Sun lawsuits.  Most people misperceive the threat that I believe Microsoft envisioned from Netscape and Microsoft’s view of Java at that time.  A common view has something to do with Windows becoming irrelevant or some other interesting but unlikely situation.  Another view that seemed to permeate had something to do with web applications replacing Microsoft Office.  This may happen at some point, but from what I can see, the company to do it will likely be Microsoft itself. Both of these still miss the core of how Microsoft sees itself as a company.

The real threat had to do with Microsoft losing its ability to control the software development landscape.  If Microsoft had lost the software development business, it would become just an OS and software application vendor completely exposed to all the other application vendors.  The interesting point from the lawsuit was that the judge had ordered that Microsoft be split into two – an OS company and an application company.  What was Microsoft reaction to this?

Ignoring the legal reaction, the technical reaction appeared to be the development of MS.NET, which can be thought of in non-technical terms as a layer between application development and the OS.  Prior to the lawsuits, it appeared that Microsoft was embracing Java as a future software development path.  Some believe, maybe correctly, that the intent by Microsoft towards Java was to “embrace, extend, and destroy”, but I think this may be a mistaken view.  The extension Microsoft made in its J++ product definitely had the impact of making the applications solely usable on Windows, but they were necessary to make the applications usable at all.

Along with being a Visual Basic developer, I also have multiple years of experience in Java development.  I understand that what Microsoft was doing with J++ was required, because otherwise GUI J++ applications would have required using the Swing library, which is a dog.  So Microsoft had the choice of releasing something that would have been cross platform (but absolute crap) or to build their own non-crappy GUI library.  The interesting thing is that you may be able to get away with using Swing now because of the significant improvements in computing hardware, but then (1997 era) it was not acceptable for business use.

Getting back to the point about MS.NET, after the lawsuit, Microsoft dropped all real attempts to continue down the Java path and threw its weight behind MS.NET.  Even if the company was split into two, it would still control the underlying foundation for its development tools.  Microsoft adding a Windows GUI library to Java did not kill Java for Windows applications, but the end result of MS.NET definitely has.  I don’t see too many Java based (non-crappy) Windows applications floating around anymore.

Louis Gerstner (IBM CEO from 1993 to 2002) interestingly says that IBM was the Microsoft of the 1960s-1980s with the IBM 360 Mainframe dominance.  It sounds interesting to me to call IBM the Microsoft of the past because I tend to view it in the opposite light with Microsoft (partnered with Intel) as the new IBM, or at least as an offspring of IBM.  Microsoft, as the child of IBM, probably rebels in some areas, but picks up where big blue left off in others.  From my perspective, it is obvious that Microsoft is much better at building development tools for small to medium size businesses then IBM was.  Unfortunately, Microsoft actually writes worse documentation then IBM did, which I thought was impossible based on IBM’s history of generating worthless documentation.  The IBM of the past was the master at writing technically correct but unusable documentation.  It could tell you the name of a switch, describe its basic function, but not tell you why you should switch it.  Instead of going down a Microsoft or IBM bashing route, let’s get back to the main point.

In the late 1970s, IBM was the unquestioned dominant computer company.  In the early 1980s, a small (maybe considered inconsequential) decision by IBM led to it’s almost bankruptcy only a decade later with Microsoft and Intel picking up the pieces.  Most leading businesses share market dominance with multiple other businesses, but in the technology game it is winner take all.  Only in technology will you see people willingly using products that they hate out of necessity.  The network effect in technology causes gravitational market shifts to occur.

The more times a program is run successfully does not decrease the chance of failure

A common mistake that is often made by businesses is to trust software systems that have run for a long period of time.  These systems are usually considered trustworthy because all known issues have been “shaken-out”, which means that all problems related to the business have been corrected.  Systems that become a tried and true piece of the business may become the Achilles' heel.  The point that businesses miss is that the longer a system runs, the greater the likelihood that the system architecture will fail.  There are multiple reasons for this, but two of the main reasons are as follows.

When software systems are built, there is a tendency to overlook the lifetime of the system in the design of the system.  All software has a point of failure, but it is rarely determined or taken into account when building systems.  When this point of failure occurs, it can lead to a catastrophic failure causing the system to no longer function, which is the best case scenario.  This is the best case scenario because the worst case is that the system continues to appear to function properly but is actually destroying your business data.

The second reason for failure is that systems are altered over time, increasing the chance of failure.  Rarely does the person altering the system have the same understanding of its function as the initial development group (I may be falsely assuming that the initial development group was competent).  These changes are usually attempts to correct some mistake in the business process logic, but sometimes it can be changes to the underlying architecture driven by some business need.  Taking one brick out of a building is not likely to cause a problem, but deleting even a single line of critical architecture code can yield dire results.

Are businesses doomed to have their critical software system fail at the worst possible time?  Not if they take action to mitigate the risk association with system failure.  Unfortunately, this requires an incredibly difficult step – businesses must change from being risk adverse to risk conscious.  As described earlier, businesses mistakenly tend to trust systems that have run for a long time without major failure, but businesses should never trust their core operational systems!  There should always be a fall-back system that can be used if the core system fails.

These fall-back systems should be able to run the business in case of the main system failure, but they can be significantly simplified to meet only the critical business process needs.  These systems also have the added side benefit of being proof of concepts for redesign of critical system.  Instead of critical systems being tied to one specific implementation, the business will have other options which can be leveraged in the future.  The critical point to understand is that creation of system business process logic is expensive, time-intensive, and risky while the designing and architecting of small software systems is not.  Businesses should assign a small portion of their staff to constantly research how to improve their critical system along with building secondary backup systems for all critical business systems.

- Luke 

The time has come for reverse direct marketing

Have you ever gotten something for “FREE” only to have to pay for it?  If you have a cell phone or satellite TV package, then chances are that you got the hardware for FREE, but now have a two-year mortgage on the equipment that must be paid or else the FREE equipment now becomes very expensive.  Why do we allow the use of the word when the product or service is clearly not free?  When I see the F-word in an advertisement now, I automatically think of another four letter word – SCAM.

What if you told your employer that you would work for FREE*, with the small condition that they sign a contract paying you 5 times your normal salary spread over the next 24 months?  Also, you allow them to fire you at anytime, just as long as they give you the next six months of salary immediately.  I would not try this as it is likely that your employer may truly set you free.  What is it about the English language that allows for it to be so easily misused?

FREE does not mean that something is without financial cost, but instead FREE means that the receiver has no current or future responsibility to anyone for anything.  This is what distinguishes FREE from GIFT.  Gifts are to be exchanged, so the giver of a gift usually assumes some future reciprocation.  Because of this, people receiving gifts can sometimes feel bad instead of good (because they are now burdened by a future responsibility).

Information, especially personal information, should not be considered FREE.  Someone giving you something for FREE*, but only requesting your personal information is likely doing you no favor.  How much is your information worth?  Also, could your personal information actual cost others money?  After listening to an otherwise uninteresting audio book about personal economics, I remember the author discussing the costs that charities incur in direct marketing expenses.  His opinion was that people that intermittently give small amounts of money to charities would actually save the charities money if instead they asked to have their names removed from the mailing lists.

Building a computer can be like riding a roller coaster, just don’t go off the rails!

Building a computer can be scary, especially the first time.  I was fortunate enough in my early career to get paid to fix computer software and hardware problems.  This did not stop me from experiencing fear when building a computer for the first time with my own money, but it did make me confident that I would succeed or at least know that it would be a valuable learning process.  Now when I build a new computer, it feels almost likely being on a roller coaster.  The emotional payoff comes when the computer boots for the first time, which unfortunately does not always happen (e.g. I built a computer with a friend and grounded out his motherboard accidently and it would not boot; luckily it did not cause any damage and we resolved the problem with some risers).

This initial boot up is like the feeling after the first major drop of the coaster.  Your anxiety lessens and you start to feel good, only to be thrown against the side of the coaster seat, which is equivalent to installing an operating system on the computer.  Both are painful events which may make your back hurt or your palms sweat, but are necessary parts of the experience.  With the coaster, the pain continues with other unexpected tight twists and turns. With the operating system, the necessary programs which allow the hardware to interact with the operating system are what cause the pain.  These programs are called device drivers and can lead you from ecstasy to hell when building a computer.

It is important to know up front which pieces of hardware will work with your operating system of choice while still working properly together.  You can skip this step if you are a thrill seeker, but otherwise I would suggest some minor web surfing to view various suggested configurations.  The most critical decisions revolve around buying the right motherboard, processor, memory, video card, case, and power supply.  These components should be considered with more importance then storage devices, input devices, monitors, and all external devices (printers, scanners, etc).  You may notice information on over-clocking the CPU and crazy cooling devices; it may be better to steer clear of this information because it is intended for the experienced computer builder who is willing to risk burning out the processor.  If you are comfortable building computers, then over-clocking may become a natural next step when trying to maximize your performance per dollar.

Be very careful when picking out various components as the top components are much more expensive then they should be and good components are much less expensive.  Fight the urge to buy a $500 video card or processor unless you just happen to have buckets of cash.  It is almost impossible to tell the difference between a $150 video card and a $500 video card anyway.  The processor is a little more important and impacts everything in the computer, but never buy the top of the line because it will be at a significant premium.  Lately, the sub $300 price range seems to be the best price per performance for processors.

When building a computer, the first and core components are in this order of importance:

1)      Motherboard – The heart and soul of the computer; a cheap motherboard will ruin your computer building experience

2)      Processor – The CPU selection used to be less important but is becoming more important with new video/audio encoding software

3)      Memory – Cheap memory can lead to random blue screens of death

4)      Power Supply – A cheap power supply can lead to random freezes of the computer

5)      Video Card – It is important to get a good video card, but do not spend to much on it

6)      Case – If you buy a cheap case, make sure to have band-aids ready as you may get cut

To research motherboards, I will usually go to a site like newegg.com or pricewatch.com and review what is available and at what price.  Then further research can be done at tomshardware.com, cnet.com, or any other trusted review sites (search on motherboard reviews).  Understand that even though a motherboard may be recommended, there may be various revisions of the motherboard, some which require updating the BIOS to correct for problems found after the initial release.  You may wish to buy an established motherboard that does not need a BIOS update for your first computer.

If not building a gaming machine, then there is no need to buy separate video and audio cards because many motherboards have decent video and audio cards already built in.  I prefer to have a separate video card, although having the card built simplifies the building process.  I bought a video card one time that was not compatible with a motherboard but was able to swap it with a new card in another computer.

Some people choose a processor first and then a motherboard, which may work for you, but rarely do I need anything which is processor specific.  If you are going to be doing anything related to video or audio, then you may wish to focus on the processor first also.  The currently available processors are incredible, so I usually cut costs from the processor purchase by avoiding anything that was recently introduced.  There is a significant price premium for the top of the line processors, not because they are that much better, but because early adopters tend to ignore price.

When choosing memory for the computer, do not go cheap.  Cheap memory may appear to save you money but comes at the cost of system stability.  Cheap memory can cause random freezes and blue screens of death.  Again, do not spend too much on memory also, as you are unlikely to notice a recognizable performance difference.  Save the money for something more valuable like a power supply.

The power supply is the component that people tend to overlook.  The better the case that you purchase, the less likely it is that it will come with a power supply.  This may not initially make sense, but what it means is that people who buy quality cases understand that purchasing a quality power supply is also important, whereas people buying crappy cases do not care about the power supply quality.  Don’t let a bad power supply spoil your experience; spend the extra $50 to get an awesome power supply.

The computer case appears to be a stylistic or aesthetic decision, but if you buy an inexpensive case, make sure to buy plenty of first aid supplies to handle cuts and scraps.  Also, the cheaper cases run hotter and are heavier, so only go cheap if you can not afford a decent case.  The case decision completes the core of the computer, but the computer is still not usable.

Yelling raises the arousal state and causes the topic to be remembered

I have never yelled in my life (well maybe a couple of times) until I had kids.  I sometimes find myself yelling at my kids when they are about to do something dangerous.  If my kids are about to do something incredibly dangerous, and I am not close enough to physically restrain them, then I will yell at them to stop.  Does yelling work?  Why does it work?  When is yelling inappropriate?

Yelling appears to raise the arousal state of those around (release of adrenaline), which allows for others to remember what is being communicated.  A message to not run into the street is a message that should be communicated forcefully or otherwise it will not be remembered.  A problem comes with yelling a meaningless message or yelling too often which will cause the listeners to tune out (i.e. no more adrenaline releases with the yelling).

Yelling also has the effect of making the person yelling feel out of control when the message is either not heard or ignored.  Someone yelling at a computer is not likely to feel better afterwards, but instead will feel powerless.  If it continues, then instead of the yelling being an attempt to communicate something important, it turns into an act of aggression.  Out of control yelling may turn into rage which then causes some action, usually bad, to be taken.

People who follow this pattern even yell at the other person or thing for causing them to be angry - “This stupid computer is making me so angry!”  If anything, the person or thing is just the trigger for the anger and not the cause.  I write this even after suffering through the anger of having to learn a new version of MS-Windows for the 20^th time!  I know that the new version of Windows is not the cause of my anger, but just the trigger.  Besides, Microsoft knows more about what I want then I do, so I should relax (they have research to prove it).

When I type “c:” in the command window and press enter, I do not actually want to access my hard drive, instead I want some search process to start, randomly pick an option, and then execute it.  This is a feature that I have been requesting since the MS-DOS Shell days.  It is about time that Microsoft started listening to me and putting features like this into their operating systems.

Interestingly enough, I have never yelled at a computer, because since SmartDrive was included in MS-DOS, I knew I was not in control anymore.  Prior to that point, the computer only did something if requested.  I was in control until SmartDrive showed me that I was stupid.

Digressing from the unintended Microsoft references, I feel that yelling is only appropriate when the message is critical, otherwise I will seek an alternate method of communication.  Is this how the “evil eye” was invented?  Also, I have never seen yelling as being effective in a workplace except to cause the employees to leave.

Luke

Can you bring your laptop and work on the project at your father’s funeral?

Domain specific languages can severly limit business discussions

The first months of a new job can be the most difficult just based on the lack of understanding of the domain specific language of the business.  These specific languages are cryptic and rarely have been built with any forethought, but instead have grown from a need to quickly express ideas in the fewest possible words.  The unfortunate side effects of these shared languages are that they become a limiting factor on the topics which can be discussed and can cause significant confusion when the language is not shared (or worse partially shared) within the entire organization. 

When individuals are able to speak using a domain specific language, but then are unable to translate to plain language, their actual understanding of what is being discussed is limited.  The unfortunate effect is that these languages can be used for the purposes of group conformity or expressions of power instead of conveyance of any real meaning.  This unfortunate downside of domain specific languages can be limited by taking the power out of the language through formalized written conventions.

One of the roles for information technology groups is to document how the organization does business and generate accessible information that can be a basis for communication throughout organizations.  Simple topics can be discussed in enough specificity so that communication between groups can have contextually implicit meanings without confusion.  For example, the organization’s clients can be described differently based on their relationships to the various parts of the organization.  This triggers the various groups within the organization to define a simple concept such as a client with different understanding.  The marketing group may implicitly consider a client to be a “potential client”, the customer service group may only focus on a “prior client”, and the rest of the organization may be focusing only on the "currently active client".  This language impedance may lead to confusion and delay (Note: Thomas the train reference).

When information systems are built, they should be designed with the full understanding of the fundamental conceptual basis that the organization is using for various concepts.  This allows for the systems to be organizational systems instead of departmental systems.  Potential future orders and past historical orders should be available data for any group in the organization that requires this data for part of their business process without requiring massive integration efforts.

Once information is created which discusses the business concepts, it must be given to new employees during their first few days of employment.  Once employees began to interact with others within the organization, they will be less likely to read this type of documentation and more likely to take extra long coffee breaks.  Also, an organization which has formalized control over its own language will likely be in position to be understand its own operating processes better, making the organizational processes improvable (e.g. through six sigma or some other process improvement strategy).

Luke

Publish the NIE - At least then it would be complete instead of just the politically motivated leaked information

Should the intelligence community just publish the National Intelligence Estimate in the New York Times?  It is going to end up there anyway in a form that may be substantially different than the original causing mass confusion and hysteria. Instead of the politically motivated leak-driven flow of information going to the American people, it would be the actual estimate in its entirety.

Why not just turn all United States Intelligence into blogs and publish them on the web?  Or even better, a Wiki, because how could it be worse than it is now?  Maybe we can go back and correct intelligence that was clearly wrong in the past ourselves.  We all know somebody who knew someone who did something, which would apparently make all of us covert intelligence operatives.

I am going to contact the CIA right now to tell them that their Wiki is not working.  I tried wiki.cia.gov, but to no avail.  Also, the video and Images that they posted on Google do not seem to be working properly.  If the CIA used YouTube instead of video tape, then we could re-edit them and add our own music or something.  I have some Spice Girls tracks that would do the trick.

Oh wait!  ABC News has just gotten leaked information about the CIA infiltrating Wikipedia!  As luck would have it, there is now a counter-intelligence action being undertaken by Cal Tech to counter this abuse of power by the CIA.  Why doesn’t the CIA just stick to what it does best, like toppling democracies, stranding agents in foreign countries without hope of rescue, and giving PR advice to presidents?

But the 9/11 commission changed all of that.  Now the President meets with someone else who then meets with the CIA director.  The direct meetings with the head of the CIA were never going to work anyway.  The President can get better actionable information from the newspapers because reporters are better at piecing together bits of meaningless information quickly than any CIA analyst could even hope to be.  In fact, most reporters are so fast that they can write the story even before the facts are obtained!  Trying to get this efficiency out of the federal government is never going to work.

Luke

Software knowledge tends to be inversely proportional to hardware knowledge

Where is the performance increase in software that would be expected from the drastic increase in computer hardware speed and capability in the last decade?  Is it even generally known how fast software should be?  Most people seem to ignore the fact that software has not gotten any faster as the hardware has improved.  From running applications on the latest version of Windows, it appears that the faster hardware is actually slowing down the software!

Computers are now capable of processing gigabytes of data per second and yet most people are satisfied with software if it can only process hundreds of Kilobytes per second.  Why is this order of magnitude drop in performance acceptable?  Computers have crossed the boundary between being technical novelties with every inch of their capabilities being studied to being commoditized ubiquitous appliances, not unlike a toaster.  Maybe the problem is that people now view computers much like they view a toaster, a known device that is not expected to get better overtime.

The computer I am using now is about three years old, but its abilities would crush hardware used only 15 years ago to host enterprise databases!  The enterprise hardware systems in the early to mid 1990s were able to process hundreds of Kilobytes per second.  Most of these systems only contained a few processors (or only one) which today would be considered only usable by a calculator and hard drive storage capacity that would not be considered acceptable by a teenager purchasing an iPod.  Server hardware of today is not even comparable to server hardware from as little as 7 to 10 years ago.  Even so, the software being developed today does not capture this drastic increase in performance.  Instead, the software eats up this performance like a hungry hippo (as aptly described by a DBA from a previous employment).

So what is the problem with new software systems?  Because hardware previously was only capable of processing large sets of data in batches, it was a normal practice to design the hardware and software around this batch concept.  Hard drives are incredibly fast when used to access large amounts of data sequentially but orders of magnitude slower when accessing small amounts of data randomly.  The overheard to execute a query on an enterprise database management system does not change significantly when accessing one record or a thousand records.  Also, the distance between the end user receiving information and the actual server processing that information has grown to become like the Grand Canyon.

The physical location of these servers in the 1990s was on the same network as the users of these servers, and the servers were owned by a single group.  Now, it is unlikely in medium to large organizations that the server will be on the same network and it is significantly more likely that the processing power will be split amongst multiple groups, if not the entire organization.  The reason for this splitting of resources, while being correct technically, may be destroyed by the software side of the equation.

The big push to client server systems in the 1990s was partially an attempt to use server resources more efficiently.  The big push now is for the more effective use of technical personal.  The impact of this change is easily noticeable by technical people who have been in the industry a long time.  While the knowledge required to develop software systems has increased there has been an unfortunate inverse affect on the hardware knowledge of software developers.  This has lead to software focused on meeting business requirements solely without taking into account that the software actually has to be executed on a computer.  These systems are murdering the performance of the underlying server systems.

Why are organizations not demanding that the software architectures be proven out before spending millions of dollars on the development of the entire system?  Why are organizations accepting transactional performance from their core operational systems that would have been considered unacceptable in the previous decade?  Why is it no longer a requirement that software developers understand the underlying hardware they are using?

My experiences in IT have been that organizations only fully consider performance of software systems when implemented in a production environment and only as a last step in the process of development.  Conjectures about return on investment, meeting business requirements, and having logos that spin are usually considered with more importance than system architecture.  This lack of focus on architecture along with organizational politics that requires parts of the system that are dependent on one another to be designed and coded separately cause the complexity of software code to be significantly higher than the minimum complexity required to build the software, with each line of code added conspiring against the productivity of the organization.

Add to this mess the structure organizations are placing on software groups with management by time and cost based accounting principles, and you understand why the significant problems that exist with the performance of software systems are not able to be resolved.  It is not reasonable to rearchitect a critical production system while it is running your business, even though there are development methodologies that try to remedy poor design decisions during development with incremental fixes later.  Understand that if you actually tell the business managment what you are doing (i.e. incrementally redesigning the critical production system while it is in production), the initial reaction by management will be to determine if you should be fired.

The worst part about this whole situation is that architecting software systems is simple as long as you understand not only the software but also the hardware.  Understand that the initial technical requirements for a new system should not be tied to the business use of the system, except for the requirements relating to system performance and load.  If the system is required to process 50MB of data per second, respond to all user requests within a certain time period, and store data in a certain database system – this may be enough of a requirements specification to begin prototyping various architectures for consideration.  Businesses that put off all coding until all “business requirements” have been gathered are shooting themselves in the foot and will be surpassed in the long run by peer organizations that do not allow their IT systems to strangle their business.

Luke

 Do not let the technology get in the way of managing the Information

Most members of Information technology (IT) groups view their group’s role as bringing technology to the organization.  This misses the entire point of IT, which is to bring information management to the organization.  If organizations could manage their information assets without technology, then most would.  Technology is an expense to the organization, but that expense needs to be offset by the value that IT can bring to the organization.

What does this mean to individuals working within IT groups?  You should view your role in the organization as a member of the group that collects, maintains, analyzes, and disseminates information for the benefit of the organization.  Even though your technical skills are required as part of your employment, they are only secondarily important to the organization.  So, what non-technical skills should you learn to add value to your organization?

The most important skill is the ability to read with understanding.  Most members of IT have convinced themselves that reading is a waste of time, but this is mostly due to what they have read in the past.  The vast majority of documentation written within organizations is useless and should never have been created.  If you have read enough useful information, you will gain the ability to immediately determine a document’s usefulness.  Also, you will recognize the significance or insignificance of entire categories of documentation.

Once understanding of a topic attained, it is important to be able to retrieve this understanding coherently.  The ability to communicate with others vitally rests on top of your ability to connect your knowledge together and express that connection in a logical manner.  Have you ever gone to a long meeting or presentation and not understood what was being discussed?  The assumption may be that the presenter does not know the topic, but another alternative is that the topic was not presented coherently.  There is also the opposite issue when a presenter is an excellent communicator but does not understand the topic.  The audience may feel great about the presentation, but in retrospect have learned nothing.  This is why I avoid listening to most motivational speakers.

If you have gained the ability to receive information and then communicate it with others, then the next step is to be able to manage information, which is the critical step that makes information useful for organizations.  Without this ability, information actually becomes an impediment to the organization’s ability to understand itself.  Information overload is a critical problem facing many organizations.  Not only may there be an over abundance of information, but that information will probably be either too raw to be useful or too processed to be meaningful.

This leads to the next step, which is analysis and aggregation of information.  Most raw information is not useful to the execution of organizations by upper management.  The information’s value is significantly enhanced if it is analyzed, aggregated, and summarized into actionable information.  It is important that this actionable information actually reflect the facts of the situation without leaving out relevant information.  Information can be slanted accidently or intentionally to misrepresent the organization’s situation.  The information analyzed must be complete, conclusive, and statistically meaningful and the summarized information must yield all logical results that could stem from the analysis with justifiable reasons for discounting obviously false conclusions.

It is important to build upon your abilities to understand and communicate the additional skills necessary to further analyze and correctly summarize information.  Analysis skills do not always come naturally to most people in IT because it requires blurring the facts of the situation to generate an overall higher level view.  The mode of most IT people is to do the complete opposite – take blurred information and generate concrete facts.  Also, an additional key skill is required, which is the ability to generate conclusions from the information along with additional information the supports, refutes, and attempts to explain all other possible reasonable alternatives to those conclusions.  It is necessary to grow beyond a simple technical basis for your career because it may be the employment differentiator in the future.

Due to the increase in inexpensive storage, organizations are collecting large amounts of information.  Enough information is being collected that statistical analysis can be used to model parts of the organization and allow for simulation of alternative actions by the organization.  This is allows organizations to statistically predict the likely outcomes of actions, giving guidance that would not otherwise exist.  A problem can occur when the organization is not collecting enough useful information or has not properly aggregated that information to generate a useful empirical statistical model.  It is important to understand that empirically generated information should only be used for guidance because it can possibly lead to erroneous results for situations which are outside the norm.  If there is only a 1% chance that an event will occur, but that event would be catastrophic, then it may be reasonable to prepare for it anyway.

Built upon all other skills, the ability to handle information in the present and past can be leveraged to generate reasonable inferences into the future.  Headlights allow drivers to drive more safely at night by allowing for further sight then would normally be possible, but in no way do they stop accidents from occurring.  The ability to model organizational processes is similar to headlights on a car, allowing for insight into the future but not perfect insight.  The abilities to understand statistics and mathematical modeling are the final skills necessary to complete your training within most organizations.

If the information has been collected, maintained, and analyzed, then the final role for an IT group is to disseminate that information.  Within the bounds determined by the organization for securing the information, the information should be made available to the organization in a manner that allows for user driven access to the most current information.  Information that is necessary for the operation of the business should be made as a transactionally consistent part of the collection of operational data and available immediately to the operational groups.  Most non-operational information is not immediately necessary, but may be used by the management to determine the direction of the organization.  It is critical to have processes in place that treat distribution of non-operation data in a manner similar to a publishing effort.  Information that has not been validated should never be allowed out of the IT organization without clearly marking the information as being subject to change.

The final knowledge necessary to complete your IT journey is that of understanding organizations.  Once you realize your position within the organization is not the same as your understanding of your role within the IT group, everything changes.  IT roles are usually task based and time limited, and unfortunately the management of organizations tend to view IT employees as being useful for certain tasks and time periods.  This may be why outsourcing of IT work is so prevalent, even though most organizational management task appear to be not as critical to the success of the organization as the actual work being done and may be more easily automated.  Since salaries for organizational management is not likely to impact the bottom line of the organization and improper management can cause organizational failures, it is improbable that organizations will attempt any significant management automation anytime soon.  It is much easier and more reasonable to hire an experience manager then attempt to build a management system to do the same task.

In continuing the prior car analogy, the organization’s management is the driver of the car whereas the IT group would be considered part of the systems that collect information from the engine and presents it to the driver.  The driver of the car does not need to know or even care about the operation of the vehicle beyond that which is necessary to control the vehicle.  Drivers must maintain the car or it will surely fail within a short period of time.  The basics of maintaining a car can be extended to an organization, such as filing up with gas, regular oil changes, minor and major maintenance, but I will spare you the boring extension of this analogy.

Understand that no matter how much IT members joke about the outsourcing of management, the organizational cost of management is at least an order of magnitude smaller than the cost of the other parts of the organization.  The management cannot make the organization succeed, but it can lead it in a direction were success is a logical outcome.  However, the management can drive the organization off a cliff, as was witnessed by many examples in earlier part of this decade.

If IT groups are providing significant value to the organization, then why has there been such a push to limit the scope of these groups within organizations, and if possible, outsource them entirely?  I believe that IT groups are overly focused on technology and minimally focused on the real value to organizations, which is information management.  When purchasing equipment and software, IT groups should focus on reducing cost by purchasing the least cost hardware that meets the requirements and meeting the base requirements for software first before expanding into any features.  But this does not actually go to the heart of the matter entirely.

Organizations which do not understand or appreciate the value of information management also may see IT groups as just a necessary expense to the organization.  If the IT group is not positioned to be an asset to the organization, then what is the point of having it internally?  The main problem with organizations giving up on having an internal IT group is that now some external entity is managing the organization’s information. That external entity not only manages the information for the organization, but also can analyze that information, learn from it, and externally disseminate it without your knowledge!  Do you think some non-disclosure agreement is going to stop an external entity from learning from your organization’s information?

Somehow, I do not think Google will ever have this problem within their organization, even though they are more than willing to host your company’s data for free!  Somehow, I do not consider a couple thousand dollars worth of free storage and hosting services a fair exchange for my organization’s information assets (although I am more than willing to host other information there).  But moving information managment to an external organization may sound logical to an organization which is trying to stay alive through reducing its expenses.

Not to belabor the point, but you are responsible for yourself throughout you career and should continue growth throughout the many possible avenues of IT.  As a member of the IT group in your organization, it is also important to understanding the positioning of your group and its impact on the organization.

Non-empirically based software system development

Throughout my career as a software developer I have felt that there was a difference between my behavior and the behavior of the other developers, some of whom were significantly more intelligent and/or educated then I was.  Either recent clarity of thought or some misguided arrogance has led me to a reason for this feeling.  A description of the behavior may be useful or at least interesting to others, and I will attempt to explain the difference in my behavior from that of others which should give further insight into how I have approached learning complex systems throughout my career, and hopefully it will help others to do the same.

When assigned to a completed project, I have always broken it conceptually down into multiple distinct parts and then attempted to understand each of those parts fully.  When the understanding of a particular part has become complete, I move onto the next part.  After a full understanding of each part, I then step back and look at the project or system as a whole.  This has led to conclusions that were not in line with other developers and managers, but since I did not possess the abilities to effectively communicate these conclusions, I was unable to convince others of the validity of the solutions.

The method is simple and can be explained with an equally simple story.  Imagine that a good software developer walks into a room and flips a light switch.  The good developer will overtime (or maybe immediately) accept that the light switch controls the light.  Upon entering the room and flipping the switch, if the light did not come on then the good developer would also know from practical prior knowledge that the most likely causes of the problem are that either the light bulb is burnt out or that the power is out, both situations that can be immediately remedied or explained.  If the problem with the light switch - light bulb system is instead something other than the immediately recognizable problems, then the good developer is likely to be stumped and waste large amounts of time studying the problem with the desire of only understanding how to solve this immediate problem.  Upon finding the solution to this particular problem, the developer will add that solution to a “bag of tricks” that can be used in the future, but ultimately if another problem arises in the system, additional time will be wasted on a restudy of the system with the new specific problem in mind.

Now continuing the story, if a bad developer walks into the room and flips the light switch and the light does not go on, then the bad developer may similarly to the good developer recognize the immediately obvious problems but will not always take action to remedy them.  Only if required as part of the job and under pressure will the bad developer change the light bulb or recognize that the power is out.  Otherwise, the bad developer will just jiggle the light switch to determine that the light switch was actually turned on and then leave the room.

The truly egregious developer will come into the room every day and jiggle the switch expecting that at some point the light will come on, either assuming that problem will fix itself or become enough of an issue that another developer will fix it.  I have no advice for switch-jigglers other than to keep doing the jiggling so that we non-jigglers feel better about ourselves.

To complete the story, the first action I take when walking into the room is to completely rip the switch out of the wall, examine the switch to make sure it is of reasonable quality and is properly connected to the electrical system.  Then I follow the electrical line to the light fixture, examining the electrical line to make sure it is of sufficient quality as to not catch on fire if the maximum current runs through it.  Then I rip the light fixture out to determine if it is of the proper quality and examine the light bulb to make sure it is of the proper rating.  Obviously, if I were to do this with an actually light switch – light bulb system, I would be electrocuted and then fired for destruction of physical property.

[NOTE: after writing about jiggling a light switch, I searched on Google to see if this was similar enough to another story as to be misconstrued as plagiarism.  Strangely, there is an enormous amount of time being spent by people jiggling light switches.  This may have been what led to rocker switches and motion detecting light switches as a defense against all this jiggling.]

While software systems are the same as physical systems in that they are made of subparts which together work as a whole, they are different in that they can be broken apart and studied without damage to the system.  To understand the system as a whole, it is necessary to understand all or most of the subsystems.  I do not fully understand the working of the electrical system coming into my house, and if the power was out, I would not further study why the light switch was non-functional but would instead consider the power outage as the cause.  But, if there was a different problem with the system, no further study would be required to understand the system, only troubleshooting and likely part replacing.

In describing the light switch – light bulb system, an attempt was made to explain my behavioral patterns when assigned to complex projects in the past, mostly without me being fully cognizant of reasoning behind it except that it was a pattern of behavior that had worked previously.  My educational background has been in engineering and for a large portion of the education was in the engineering of physical systems (aerospace engineering and computer engineering).  The knowledge of physical systems can be used when creating and analyzing software systems, but it is also important to understand the differences.

Fully understanding the physical light switch system is not important to most people as the system was likely built with reasonable quality due to building code enforcement and is used so infrequently in comparison with software systems that its breaking is almost inconceivable.  Software systems, on the other hand, are made up of an incredibility large number of similar simple systems, which are rarely built to any consistent standard and are used so frequently that virtually all conceivable problems with the system are likely to occur with its use over time.

Recognizing my ability to solve complex technical problems quickly using this method and the fact that I was unable to communicate what I was doing with others led me to spend the last couple of years outside of my software development work practicing my written communications skills.  My first attempts at writing were uninteresting and would not have been meaningful to others, which was a point of frustration as without the ability to successfully communicate information in written form, I do not know of another form that can be used (e.g. non-verbal communication, my primary communication method, does not work well when communicating technical information; I know this from watching multiple hours of the three-stooges as a child).  Even so, these first attempts at written communication led me to the belief that gaining this skill was an important, if not critical, next step in my growth.  Something I wrote in my earliest poorly written writings was that there was little purpose to understanding without the ability to communicate that understanding with others.  From that point forward, understanding was no longer the end, but now was the beginning.

Luke

Overall database performance is a function of data size alone

I was reading an article online which stated that all database performance issues are caused by applications.  It was explained that the database did not have a performance issue before the applications connected to the database!

Software developers view a database’s performance through the requirements of their process or application but ignore the overall performance of the database.  At the same time, database administrators view the database as a whole and rarely fully understand the needs of the users of the database.  This is the root cause of battles between developers and database administrators, and the likely reason that database schemas become worse over time.

Like a one trick pony, if the database is used by a single user and will never have any additional requirements, one trick may be enough.  There are many little databases spread through organizations which should never be considered for general performance optimization, but the databases running the critical systems of the organization should not be one trick pony systems.  These databases should be designed, development, and maintained as critical entities themselves and should not be tied to a single user, application, or process.  The design of these databases should instead dictate their usage by external systems.

What this means is that the entry of data, lifetime of data, and removal of data from the database should be considered as part of the requirements for the database design. Without this requirement, almost all performance related database schema alterations will become unnecessarily difficult.  Databases should not abdicate the responsibility of data management to external applications or processes by allowing external processes to interact directly with its internal data. This direct interaction with internal data ties the database performance directly with these external processes.  I will write a separate document explaining how a database should be designed to allow for this separation, but within this document it will be assumed that the database was designed with this in mind.

If the schema of a database is not bound by direct interaction with externalized systems, then it can be modified to allow for significant system performance optimizations almost solely by reducing the data size (or apparent data size) alone.  The first thing that is asked for by process developers when attempting to optimize a query is usually the addition of a new index to a table.  Understand that adding a new index in a database with performance issues is like giving a drowning person a glass of water.  Over indexing of tables is a major indicator that the schema should be altered.

Instead of the addition of a new index, it may be more beneficial to alter the table schema.  If a field is sparsely populated, then it may be desirable to split this data into a separate table.  If a review of the data shows that two or more separate concepts are being stored within a single table, then the concepts should be split out into separate tables.  It is a common practice to store multiple sets of data with mostly common fields in a single table and then add flags which determines the precise meaning of the data.  When the table size grows, a request for indexing these flags is sure to follow, but the better alternative is to separate the concepts into independent tables, which may not be possible with externalized dependencies on the schema.

DBMS vendors have understood the data size relation to data performance issue and have included partitioning methods, which in effect reduce the apparent data size without requiring alterations to most processes that interact with a table directly.  Partitioning can be a valuable tool for database administrators to increase the overall system performance, but attempts should be made first to maintain a well designed table schema.  Reducing the database size through the appropriate schema design is the first and best option for overall database performance.

Luke