Search This Blog

Tuesday, December 15, 2009

Christmas Wish List

The 13+ years I have spent in Healthcare have sensitized me to some things. I might have preferred to remain ignorant of many of them. In the spirit of Christmas, which is handy at this time of year, I'd like to nominate a list of gifts that would bless all residents and citizens of the U.S.A., regardless of theology or philosophy. Each item in the list relates to health care.
  1. I wish that the role of technology could be clearly understood. There are a vast number of well-funded voices who want us to think that technology is health care or that health care is technology. In reality, technology is best thought of as a tool--an inert and often expensive piece of equipment, which, in the right hands, can produce wonderful results.
  2. "The Patient" or "our patients" is not the same as "my patient" or "Josie Jones, patient". It may not be possible to apply technology designed for delivering care to a generic patient to Ms Jones. That doesn't mean that the technology is bad. It only means that the technology must allow for deviation in procedure. I wish that the role of abstraction is system design could be clearly understood.
  3. I wish that all of the factions in the healthcare struggles were clear about their goals--with themselves and with each other. Only by being self-aware, open and open-minded can the parties negotiate a solution advantageous to all. Doctors, nurses, administrators, vendors, government and patients are currently at odds. The friction is not only between factions but within factions. Who will speak for physicians? The A.M.A.? Mayo Clinic? Who? Who speaks for government, for vendors, for patients, for nurses, for administrators? Each of these groups functions like a mob--surging to and fro as a strong voice emerges and then is drowned out. Each group must organize itself before "healthcare" can be organized.
  4. Though I recognize that individuals and groups may be driven by ego to appear more knowledgeable than the next, I most devoutly wish that each of us might recognize that the person across the table might actually have some knowledge that we don't. I wish that we would listen first and assert only when necessary. I wish that we could see ourselves as occupying the same driverless bus.

There are many more things I might wish for this year but I don't want to seem greedy. May you each be showered with blessing upon blessing as one of God's beloved and may we fully appreciate each blessing as it comes.

Tuesday, December 8, 2009

Principles of Data Governance

Malcolm Chisolm, in a recent column in Information Management (A Principles-Based Approach to Data Governance) raises an excellent point. In 2006, when I attended my first Data Governance Conference, there was much discussion around a definition of DG. Implicit in this discussion was the need for something that was concise, yet comprehensive, and on top of that, engaging. The idea was that this definition could be used:

  • As part of a sales pitch (like a slogan)
  • To create synergy within the emerging discipline
  • To provide focus for any ongoing methodology efforts

Some present may have had additional motivations, but I think these were the ones on most people’s minds.

The definition that emerged was acknowledged to be a work in progress. By the 2008 Conference, one of the tutorials quoted three definitions:

  1. Data Governance refers to the organizational bodies, rules, decision rights and accountabilities of people and information systems as they perform information-related processes.

  2. Data Governance is the practice of organizing and implementing policies, procedures and standards for the effective use of an organization’s structured/unstructured information assets.

  3. Data Governance: The execution and enforcement of authority over the management of data assets and the performance of data functions.

These were troublesome to me then, probably for the very reason that Malcolm mentions. All seem to acknowledge a context based on an organization’s information assets, but their focus seems to be quite different. The feeling I have is that they are advocating a judicial, legislative and executive approach to governance.

In the U.S., a Constitution lays out these three perspectives and establishes the mechanics (framework, architecture) within which governance will be administered. Within the Constitution and before any of the mechanical parts are discussed, in fact, within the preamble, first principles are asserted. The writers tell us that what follows will be a system of governance for the purpose of

  • Forming a more perfect union

  • Establishing justice

  • Insuring domestic tranquility

  • Providing for the common defense

  • Promoting the general welfare

  • Securing the blessings of liberty to ourselves and our posterity

While this model would probably work on its own in establishing [data] governance, there are just a couple of nuances that will have to be accommodated because our system will not be working in a representative democracy but in a corporation.

Within the context of our system, leaders are appointed and serve at the pleasure of stockholders rather than the public. The principle of one-man-one-vote does not apply. One person may control sufficient votes to dictate to the Board of Directors. Within the day to day operations, the ability to dictate policy, direct activities and appoint deputies is granted at multiple levels, though always subject to the pleasure of the higher levels.

Having now established a context, it’s time to agree on some first principles for data governance. The candidates are:

  • The entire corporation must agree to be subject to the system.
    While those placed higher may still, at their pleasure, appoint and dismiss deputies, they must agree that [data] governance operations will be a factor in those actions.

  • It must be understood that within the corporation, domestic tranquility, the common defense and the general welfare are all dependent upon the information assets owned and managed by the corporation.
    When the system is followed, all processes will flow smoothly, problems are addressed at the process level, personal antipathies are secondary to process execution and process anomalies such as unplanned rework and delays are greatly reduced in number or even eliminated completely.

  • Consistency is everyone’s goal.
    In a work context, surprises are almost always seen as negatives. Our goal must be to improve the consistency of our processes and their outputs such that surprises become exceedingly rare (six sigma has been suggested as a goal) and predictability becomes commonplace.

These principles should be the touchstone(s) of our efforts. Everything we do should be evaluated on the degree to which these principles are addressed.

I will suggest that these may also be the principles of the corporate Quality Assurance effort and remind everyone that they are also the basis of Deming’s 14 points as well as other quality improvement methods. No improvement is possible without first establishing a stable (consistent) process.

I leave you with one final principle: Data governance will not be implemented as a stand-alone initiative. If we cannot see data governance as part of a larger, comprehensive system of governance, we will not be able to address any of the three principles suggested above.

Friday, December 4, 2009

Survival, Error & Technology

I'm going to pass on some wisdom here. It's not very often that we encounter wisdom today, especially where technology is concerned, and it's often the case that we don't recognize or acknowledge wisdom until we're looking back over the wreckage and trying to figure out what we should have done. I'm probably also setting myself up by labeling this as wisdom but I am so weary of seeing the same ads with different acronyms and talking to the same people with different names.

You will never find your way out of the current mess you're in or about to be in by searching for and hiring someone with recent experience on a specific product. To put it another way, a specific product, no matter how much buzz it enjoys, is never the answer.

I will be among the first to acknowledge that the use of absolute language (never, always...) and even the use of unqualified superlatives (best, worst, fastest...) is a habit to be avoided, nevertheless, decades of experience have proven that the absolute statements in the preceding paragraph represent wisdom and that failure to heed this wisdom will produce cost overruns, timeline disasters, confusion, stress, employee turnover and a host of other undesirable outcomes.

In large part, the success of the human race has been due to our ability to recognize exceptions without necessarily understanding the rule. My own take on this is that, with today's reliance on technology, we may have reached the point where the process of natural selection that has honed this skill over countless generations has now produced a liability. "Something's different," is enough to put us on guard and may be enough to launch a complex defensive reaction to preserve the safety of the individual or group.

First of all, while it is still good to recognize exceptions, it is now absolutely essential (that's an absolute absolute) that we develop the ability to recognize the underlying rule. A study of human error (Human Error, Set Phasers on Stun...) shows that leaping to conclusions about the rule is what produces the error condition. In fact, if we can't describe the rule in terms of the logic of the computer (if... then... else...), we can't rely on technology at all.

You might ask, as I did, how we might acquire this ability. The time tested way is known as [survivable] experience. There are a host of cause-and-effect analysis tools and techniques that have the appearance of rigor and reliability and are an improvement over experience, especially when combined with exhaustive testing, but you will find that even these are more productive when used by people with experience in the world being analyzed.

Tools are great and another critical human enabler, but--and this can't be over-emphasized--no tool is so advanced that it runs itself. Every tool, no matter how advanced the technology requires human hands and a human mind to guide it. If you were to be presented with the greatest woodworking tool in the world or the most advanced sewing machine or fishing gear or computer, would you immediately become a master cabinetmaker or designer or fisherman or software developer? You might note that the only immediate change is one of expectation.

An experienced person with rudimentary tools is more likely to produce a quality result than the inexperienced person using the "best" tools. The fact that I used a tool just yesterday says nothing whatsoever about the level of my experience in producing the required outcome. I have made the mistake of looking for help and focusing too narrowly on what amounts to recent experience with the tools in my shop. I have learned (through experience) that I will enjoy better results if I'm learning while interviewing my prospective employee. If I'm talking with someone whose knowledge stops at the tool's user interface, then I had better be prepared to devote myself to directing the employee's work. If I have a staff composed of such employees, then I need to possess all of the requisite experience myself or else be prepared to conduct a project whose principle product is more experienced workers.

The challenge is to find the right mix of experienced people in supervisory or team lead roles and people who possess dexterity but are in need of experience. If I'm in a director or management role, I have to have experience producing a product with that scope. A technology "system" has a complexity that is beyond human comprehension. The only way to design and build it is through a process of identifying smaller and simpler pieces, building those and then assembling them into the final product. You need to look for people who have an appreciation for the amount of effort this takes and the discipline--both personal and organizational--that it takes.

Stop looking for Oracle or CRM or Rational or even "use case" or "data model" experience except as clues about the approach that the candidate might be expected to take. I understand that these things are ideal as targets for a logic rule processor, but the rule ("find resumes that include these terms") is so simple-minded as to be useless. If your only goal is to turn 1000 resumes into 100, then proceed, but if your goal is to find someone who can get you out of the predicament you're in, then you should spend more time on your rules so that the exceptions are more productive.

Monday, November 23, 2009

Business, Information and Technology

Are you in management? Do you have annual/quarterly goals? Will you be held accountable for achieving those goals? Is the accountability expressed in bonus dollars? Is there any possibility of a zero bonus?

Are you still with me?

How will the achievement of your goal(s) be measured? Please note here that "how" has two dimensions: one is related to process and the other to a unit of measure. In all of my vast personal experience, all of the attention has been focused on the unit of measure part (when there has been any attention at all), and the process part has never even been part of the conversation.

Please understand that what follows is not intended to sling mud at any individual or organization. My purpose is to clear the air so that we can talk about how we're really going to achieve our goal(s).

I am going to generalize based on extensive, though anecdotal, experience. In other words, I have not conducted a survey, scientific or otherwise, and cannot produce data to back up anything I'm about to say here, so I'm leaving it up to you, the reader to determine whether it feels like truth or not. Should you feel that this does not ring true, or should you wish to fault me for not being more objective, I would ask that you produce a study or at least a body of experience in support of your position.

Awards of bonus dollars tied to achievement are based solely on whether the holder of the dollars wishes to give them away or not. There is rarely, if ever, any protocol defined for defining metrics, units of measure or measurement process. You will go into an "annual assessment" meeting with your boss and he or she may discuss your level of achievement in very general terms before announcing the amount of your bonus or a recommendation for an increased level of compensation.

Why is this important, you ask? Well, it is important because it's the the way things are done. Despite vigorous protests to the contrary, the business world is set up to run on subjective assessment supporting subjective decisions. What, you say that your decisions are "data-driven" (objective)? I would love to hear the story behind the data that was used to arrive at your most recent decision.

All of this is background for understanding why "data" initiatives so frequently become mired in a swamp of politics and personality. Let's walk back from a data-driven decision.
  1. You are able to make the decision because you trust the data.
  2. You trust the data because you are familiar with and trust its source.
  3. You trust the source because you know that it is reliable.
  4. You know that it is reliable because it consistently produces information that can be relied upon.
  5. The source has been consistent because it always uses a tried and true methodology (set of processes) to produce its product.
  6. The consistency is possible because the methodology includes steps designed to validate the source's inputs.
  7. The validation decision returns us to #1.

How do you feel about standards (e.g., standard operating procedure or SOP)? If you don't currently support the creation, implementation auditable use of standards or, at any time in the past have not done so, you have no right to and almost certainly do not have access to reliable information and therefore no claim to data-driven decisions. Just to drive the point home, when your boss decides that you won't be getting that bonus or increase you were counting on, your only acceptable response is to smile and say thank you.

By the way, if you notice that your bank account (or budget) is suddenly much bigger (or smaller) than it was yesterday, what is your responsibility? Who are you accountable to? How much trust can you afford? Now you have some insight into compliance.

The use of technology introduces an additional huge portion of uncertainty into the trust equation. Take another look at the decision walk-back above and note the points where the use of technology means adding additional paths and complexity to the validation process. This is what your data governance people are trying to get their arms around.

To summarize:

  • data-driven or intelligence-driven decisions demand trust
  • trust demands reliability
  • reliability demands consistency
  • consistency demands compliance
  • compliance demands governance

OK, you can go back to work now.

Tuesday, November 17, 2009

What Is "Data" Anyway?

If you have any experience with phone support (on either end) you will recognize how easy it is to get deep into a process before realizing that the other person is on a completely different path than you are. RTFM (Read The F---ing Manual) often pops up as the answer to our communication difficulties, but it clearly is not the answer or it would have been universally embraced by now.

I happen to be a proponent of the theory that the answers aren't nearly as important as the questions. As a teacher, I know that learning is happening when the pupil is asking questions--particularly a series of related questions. This has become very important as I have attempted to make headway on [data] governance and [data] quality.

My early attempts assumed that everyone knows what data is--and they do, in the same way that a picture is worth 10,000 words. Each and every person you talk to knows what "data" is and each has a different idea in mind. For most, it's a picture of the last set of values they looked at. This might have been a spreadsheet, a graph, a collection of measurements... The key is that data is a set of values. For some, "data" is a commodity. It is files, stripes on a disk, a percent of capacity, a quantity of bytes measured in "mega-", "tera-" or "peta-". For still others, "data" is represented by a schema, model, definition, or some other abstraction.

Given those varying perceptions or perspectives, is it any wonder that at some point in the quest for "data" anything we find ourselves stuck in the quicksand of confusion. Even when all parties have been saying the same things and have been agreeing on goals, there comes a point when someone will say, "We're not going to do that." or "I don't see why that is necessary." or "But that will change my work flow." This is frequently the point at which everything starts to unravel.

So what I have learned, and what I offer to you now, is that the initial phase of any data initiative must be a carefully constructed education process to insure that the quicksand moment never happens. This must be thought of as risk management. Remember, too, that the really important questions (and answers) initially are not the ones you're hearing. All of the different constituencies are going to be much more comfortable exchanging information (or misinformation) within the tribal group than with "outsiders."

The best way to head off this risk is to carefully choose allies from each constituent tribe and use informal conversation about their pain points and the ways that data figures in the relief of that pain. You will be setting these people up to be the "experts" within their respective tribes. Part of this will be coaching them in how to respond to questions and discussion in which they don't feel themselves to be on firm ground. They need ways to postpone a response until they've had a chance to confer with other experts. This is easy to do by setting up a collaborative model.

Rule one of this model is that I never answer for someone else. Everyone can understand that a situation involves yet another perspective and that it is necessary to involve someone from that tribe in order to get a complete answer. The most common danger here is "We don't have time for that." Everyone must understand that this is an absolute red flag event. It signals that we still have not achieved a universal understanding of objective.

When a red flag event happens, it isn't the same as finding ourselves up to our necks in quicksand. It just means that we need to engage in some risk mitigation. It's a sign saying "Quicksand ahead." We will need to bring this person into the fold--usually through informal and non-threatening discussion with at least one peer or trusted expert.

Your role, should you choose to accept it, is to be a non-judgmental, constant, committed, and helpful presence that can be relied upon to be a neutral mediator and facilitator who feels like a friend in any need. Your motives must be above reproach. You cannot count on and should not hope for recognition. All around you will be better off for your presence.

If you are senior in the organization to this person, you should make sure that you are appreciating their contribution but they will appreciate non-public affirmation since putting them in a spotlight may negatively impact their ability to continue to function in the same way.

"What is data anyway?" is a question that requires many answers initially and one answer eventually. Remember, though, that many people are really only interested in what they have to do differently. "Data" may have no meaning whatsoever in their day-to-day responsibilities even though they may be monitoring real-time run charts with instructions to take a specified action when the line goes above or below a certain point. You can't possibly know where to start or where to stop in defining data for them. That's why you need the tribal expert.

Don't seek "important." "Helpful" will take you much farther more quickly.

Friday, October 23, 2009

Disturbances in the Force

So, if we know how to make things better in terms of data quality and we're motivated to do so, what's stopping us? A word of caution; what you're about to read may be harmful to your health.

Maybe you're old enough to have lived through the Watergate fiasco and can remember the facts coming to light one by one in the press until they eventually began to make a complete picture. Maybe you remember the Hollywood version, All The President's Men, in which the whole picture was produced in two+ hours rather than months, or maybe the whole thing is in the same category as the Crimean War for you and is nothing more than a question on a pop quiz in one of your least favorite subjects.

I'd like to suggest for your consideration that if we want to track down why we are having such a difficult time accomplishing something that we all claim to want, we need look no further than the paragraph above to get all the answers we need.

First, let's imagine that data quality is like truth in government. It's a good thing and we would like to assume that we have it. If, in fact we do not have truth in government (or data quality), who benefits? The answer is that it is in the interests of those who believe they can/will be blamed for the status quo to cover up the problems and subvert efforts to get at the facts that can provide the complete picture. This is especially true if they are responsible for the problems. Even if the only identifiable responsibility is that the person is the supervisor or manager of the function(s) that owns the troubled processes, they still may elect to resist and subvert in order to avoid becoming responsible for the fix.

If we want to avoid this situation, we should absolutely avoid any questions that seem directed at why or who or even how. We should avoid to the extent possible, any investigation into the past. Try to keep all discussions focused on process-based causes that might be producing the effects you are seeing. Do not zoom in on isolated instances but look for trends. Remember, your goal is not prosecution but consistent quality.

In the words of Bob Woodward's source, Deep Throat, "Follow the money." The programmer-champion will struggle against this repeatedly. There is a perception that implementing integrity checking at the point of input represents added cost. Like any other complex process, system development should seek to minimize total cost of ownership rather than any single cost line item. If it takes an extra day of programmer time to ensure that we get 99.99% consistency of integrity in the database and thereby avoid dedicating multiple full-time staff to data clean up, this is a net cost reduction.

Our system design and project management processes may not be mature enough to assign dollar values to this, but it should be easy to determine how much money we are spending on fixing poor quality data every month (or year) and then amending the design and development processes to devote a fraction of that amount to prevention.

The final perspective to be extracted from our example is that a short attention span provides little hope of even recognizing that a problem exists let alone understanding it enough to develop a mitigation strategy. Data quality (and truth in government) requires that everyone be involved. People are capable of recognizing self interest within the corporate interest and enough people will be motivated to act that the ball will be kept moving, but the media of the late 1960's is not the media of 2009. In the 60's there was an interest in the truth that perhaps doesn't exist today. In your corporate environment, you may find it easier to maintain a constant pressure of communication directed at a single theme. The widespread motivation will not be produced by a single appeal surrounded by banners and fanfare and free cake. A communications campaign must be designed for the long haul with continuous refreshing of the message.

You don't even have one percent of your employee workforce today who are ready to grapple with the issue of data quality. You are going to have to break it down in multiple variations and start with the concept of data itself. What is data? You'll be surprised at what you uncover when you go out to talk to people about their data. Stay tuned for some samples.

Thursday, October 22, 2009

Programmer as Data Quality Champion

The programmer is the one who takes all the wish lists and turns them into something that a programmable logic device (computer) can execute to fulfill the wishes. Today, several additional roles have attached themselves to this process. The architects, designers, modelers, testers... all play an important part in the final product but it is important to remember that these roles have motivations other than the ability of the product to satisfy wishes. At best they satisfy a different set of wishes that have more to do with the process than the product.

In the not so long ago days when I started in the information systems industry, none of those other roles even existed. It was all about programming.

In talking with a programmer, you will detect a hint of pride and superiority based on the sure knowledge that none of "them" could produce a program that ran without error and produced a useful result. Other than the "end user" or simply "user", there may be no one lower on the respect totem pole than the "data" people. The programmer only needs to know what you want it to "do"; data is just something that you move from one place to another.

In those bygone days before information technology there were organizations known as data processing. I'm leaving out broad segments of programming known as systems programming because at the operating system level, the data really is a commodity consisting of groups of on/off bits known as bytes. In the very act of ignoring this segment of programming we stumble over the origins of our problem. In early computer systems, there really was no data as we think of data today.

A programmer could grant wishes by making a process that took large amounts of "data" values from one file, combined them with large amounts of "data" values from other files and deposited the resulting "data" values in a new file from which address labels or paychecks were printed. The programmer's responsibility was simply to make sure the program didn't unexpectedly halt.

At first, they just told the users what the data values had to look like in order to ensure that the program kept running. When the users proved incapable of guaranteeing the necessary consistency, programmers took matters into their own hands and created scrubbing programs that would for example guarantee that a file contained only values that looked like $nnnnnn.nn where the value of n is from the set (0-9). Now everyone was happy until one day a big order came in for $1,250,000.00 and was thrown out as erroneous. At the same time, someone figured out how to divert the fractional round-off amounts into a private account.

I'm leaving out some reasoning steps in an effort to keep this to an essay length. If you get lost, just drop me a note and I'll be happy to fill in any missing pieces.

Eventually it was realized that we don't have to store data in a form recognizable to humans--the computer could be taught to present a data value in any format that a human might care to see. This leap forward allowed programmers to distance themselves even more from the data. The idea to take away from this is that programmers may not have the same concept of data that you do.

When non-programmers talk about data, they are typically talking about instances rather than types. To a non-programmer, "Walgreens" is an example of a piece of data as is "sea foam green" and "$900 billion." To a programmer, these are all character strings or text values and may be of three different subrange "types". The subrange (store, color, gross revenue) determines how the value should be handled and the value may be acceptable if it fits the pattern defined for the type.

Today, there are many opportunities to enforce patterns on data values and most of them require no programming at all. The problem is that they all produce errors and error messages that the typical user could not hope to comprehend. In effect, they cause the program to terminate unexpectedly. So, despite all the advancements in technology, we are still having to scrub data files. The alternative is for the programmer to think like a human instead of like a programmable controller and the problem with this alternative is that it introduces orders of magnitude increases (x10, x100...) in complexity and corresponding increases in development costs.

So how can programmers become champions of data quality? One relatively simple way would be to avoid accepting text values as program input. This tactic is a favorite because it defers many decisions until a later time when "we know more" and the big problem here is that we never have to go back and change it. An example here might be useful. Imagine that you are programming a system that accepts input from nurses who are taking vital signs (temperature, BP, pulse, respiration, height and weight) in a patient exam room. You take the usual shortcut and implement all the fields on the screen as text.

Everybody is happy because the nurses don't ever have to go back and correct anything and the program runs without apparent error. One day, though, a health insurance company decides to reward its contractual clients by paying a slightly higher rate to those who document that they are doing a consistent job of collecting vitals at every patient visit. Now we're asked to verify that we do an acceptable job of collecting and recording vital signs. Since the values input to a screen go directly to a database, we should have no problem. It is, in fact, no problem to count the records for which there is or is not a value in those fields, however, when we attempt to aggregate those values to show the range of values or the average value, our query fails. the aggregation query must convert the text values in the pulse field to integers and the text values in the temperature field to floating point (real) numbers in order to compute an average.

We finally discover that pulse contains some values like "100.4", "98.5", "SAME"... that cause an error because they can't be converted to an integer value. When we look at this as a nurse or physician, we can see that the mind ignores the labels on the screen and simply produces a picture of the patient based on the values displayed. Our poor computer, though, is unable to continue. The database architect could have made pulse an integer type and the DBMS would have enforced that typing by not allowing these values to be stored in the database. Using a text type allows the DBMS to accept any value for storage. The programmer could enforce a text value that is guaranteed to convert to an integer or could enforce integer types directly but in order to do so he or she must handle resulting errors in a way that is understood and accepted by the nurses.

More often, though, the nurse managers show the incorrect data to the nurses and exhort them to pay more attention. Do you believe the nurses will respond better to blame and exhortation or to assistance from the program? Check out W. E. Deming's Red Bead Experiment to get your answer.

The programmer champion will be suspicious of a discrete valued field whose data type is text. A value that may be used in a computation or any other operation where a conversion must be done must be investigated carefully. Any value that may be used as a tag for identifying rolled-up aggregations, such as store name, must get additional attention if we don't want to see quarterly sales for "Walgreens" and "Walgreen's" and "Wlagreens". The time to catch and repair these data quality errors is the very first time they are captured by a computer program. That makes the programmer responsible. Other roles have a duty to identify situations where these problems might arise, but only the programmer is positioned to do anything about it.

I realize this is asking a lot. A programmer is only human and can't be expected to know everything (right?). This suggests another way in which the programmer can become a champion. Since it isn't possible for one person to know everything that must be known (hard though that may be to swallow), the programmer must develop enthusiasm for consultation and collaboration. Every role in your environment was created for a reason and each has its own goals and responsibilities. The programmer is accustomed to the data people coming with requests. The requests are nearly always framed in terms of something that the programmer should do to make the [modeler's, architect's, steward's...] life easier and improve overall quality.

It's easy to understand how this can get old in a hurry. The solution is for the programmer(s) to sit down with these other roles and get everyone's needs on the table. All of the other roles mentioned have a different view of data than you do and here's the thing--their view is much closer to that of the customer/user than yours is. You need each other.

Accept that you are a key member of a team and as such the team can't succeed without your commitment. The flip side is that you will not be able to enjoy the success you dream of without the commitment, skills and knowledge of the rest of the team. Be a Data Quality Champion--it's within your grasp.

Next we'll take a look at some forces that act to keep the team from being all they could be. Stay tuned for Disturbances in the Force.

The Problem With Quality

  1. Meeting the spec[ification]
  2. Documented adherence to established [process] norms
  3. The product's effects are primarily positive (for example, it tastes good and doesn't make me ill)
  4. I'll know it when I see it

Multiple choice: which of the above (choose only one) is the definition you want applied to your new car?

Does the answer change if we apply the definition to your morning cup of coffee?

Last question: which definition do you apply to the next example of "business intelligence" that comes before you?

I have two points in mind. First, defining quality is not an exact science even within a specific context. Second, #4 may be the deciding factor regardless of #s 1-3. In the end, the consumer/customer merely has to say, "That's not what I was looking for" to relegate a product to the trash heap. We all know that it does no good to say, "This is what you asked for" (meeting the spec) or "I did it just like you told me" (followed established procedure) or "One won't hurt you" (tastes good and not sick--yet).

So what is quality and especially, what is data quality? How we obtain data quality is completely dependent on the answer to this question.

I'd like to suggest that we divide the question in order to produce at least one useful answer. If we examine data quality from the perspective of a computer and its logic, we can come up with an answer that will allow us to progress. The second perspective is obviously the consumer/customer or the human perspective.

Recently I received an email with what at first glance seemed like an innocuous statement full of typographic and/or spelling errors, but when I actually looked at it, it was a nearly perfect illustration of a principle I have been talking about for years.

Teh hmuan mnid is cpalbae of miankg snese of amslot atynhnig taht ftis smoe bisac ptratnes.

This is the principle that draws the line between computer logic and human "logic". It is also what makes programmers (an outmoded term, I know, but best suited to the point I'm making) so vitally important. There is only one role in the continuum of roles involved in producing an information system product that must bear the full weight of responsibility for the integrity of the data quality at the boundary between computer and human. That role is best thought of as programmer.

Unless you earn your living as a programmer, some alarms may be going off. In fact, if you are a programmer, some alarms should be going off. I earned a degree in computer science and made a living as a programmer. From there I moved into data modeling, data administration and database administration. Now I'm involved in data quality and governance. In all of that time, I have never come into contact with any training, education, book or even a job description that addressed my accountability for preserving data quality at the man/machine interface.

This may be a poor forum for this, but my intention is to change this situation right here. My next few posts will present some background on how a programmer might live up to this responsibility and some of the forces that will need to be fended off in order to make it a reality.

Next: Programmer as Data Quality Champion

Thursday, October 1, 2009

Answers and Questions

We are living in a time in which seemingly everything is driven by the need for answers. You may be saying, "So, what?" as you read this but the impression I have developed as I have moved through this time is that in many, if not most instances, this amounts to nothing more than "trivial pursuits." A quote found online recently sums it up perfectly:
"We are compelled to drive toward total knowledge, right down to the levels of the neuron and the gene. When we have progressed enough to explain ourselves in these mechanistic terms...the result might be hard to accept." [Edward O. Wilson]

Unfortunately, though having answers readily available can get you around the game board ahead of everyone else, it can't necessarily produce the big win for a cooperative venture. I don't know what research exists on this subject, if any, but it seems to me that living in "the age of information" is vastly overrated. If you think it is good to have a library of information at your fingertips then you may want to think about some desireable properties of information, for example:

  • Is it true?
  • Is it accurate?
  • Is it current?
  • Is it useful?

A relatively recent recent development has been the creation of a new area of specialization around data quality. This is not a technical post, though, so I leave it up to the reader to go as deep as necessary. The point I would like to make is that the useful property trumps all the others.

The publicity that surrounded recent preseidential elections in the U.S. shows us that "information" need not be true or accurate or current in order to be useful. If nothing else, this should cause us to pause and consider our thirst for information.

The bottom line is that a command of facts is useless unless those facts enable us to accomplish something. This is the function of experience, education, training, practice... In truth, a person who commands an encyclopedia of facts and is fluent in acronymese (see previous post) may or may not be able to accomplish a goal. This is also why we tend to look for experience when we need help.

The catch is that a person who has no experience, training, or applicable knowledge has no way of recognizing experience that will be useful. This leads to insistence on things that may have little or no bearing on eventual achievement. For example, when we need a new roof on our house, we don't ask that the roofer have experience with a certain brand of shingle or a certain kind of fastener. We don't specify how the work will be done. We simply insist on straight lines and no leaks. It's inexplicably strange that we do not take the same approach to technology goals including those relating to information. Read some recent position descriptions or postings on Monster or Dice or CareerBuilder. They are chock full of brand names to the extent that the real goal is obscured.

The proof of my point is the extent to which the information management situation is becoming more complex, with dramatic increases in the amount of work and the specialization of that work. These increases in turn produce increases in costs. It's hard to point to achievements today. Micromanagement, based on a swarm of facts with no understanding produces a lot of activity and few results.

Tuesday, September 22, 2009

Complexity and [Over]Simplification

Pardon my absence (if anyone noticed) while I recovered from a bout of depression brought on by repeated exposure to thought streams defined by buzzwords and marketing hype. Ulysses (James Joyce) was memorable for steam of consciousness paragraphs that ran on for page after page. I did not find the process of navigating someone else's thought stream enjoyable in the least. This is the same feeling I have been experiencing of late. The process of mapping the path from stimulus to response (or problem to decision) in humans is poorly understood at best.

My thought was to try to zero in on one kind of problem to see if I could cast some light on the decision-making process and find out how it gets caught in the deep ruts so often.

If you are a technophile who buys in at the bleeding edge and who speaks in acronymese (let's say it's pronounced uh-kron'-uh-meez): a language based on acronyms or sets of initial letters chosen or pronounced in the form of words, for example SQL (for Structured Query Language), pronounced see-kwel, then you may as well find something else to read because I'm about to ask you to take some real responsibility. Actually, I'm going to suggest that those who have to listen to you should demand that you take responsibility.

Many of you will be familiar with the Ishikawa Diagram though you might recognize it as the fishbone or cause-and-effect diagram. If you have ever used this tool to identify the cause(s) of a problem, you may remember just how quickly the diagram can become unmanageably complicated. A few years ago S.M.Casey wrote a book titled Set Phasers on Stun: And Other True Tales of Design, Technology, and Human Error. This book looks at some of the biggest man-made disasters of the past twenty years in an attempt to identify the cause(s).

While human nature demands that we be able to blame someone for anything that goes wrong, this research very pointedly shows that each time we identify what we believe to be the cause, it is always possible to say, "But if x had been alert, the damage would have been minimal or avoided entirely." In short, the cause is always a related set of events that may have been initially set in motion by a "proximate cause" ( The problem, if we're interested in insuring that the damage is not repeated, is that legality is irrelevant as are the needs of human nature. The only way to guarantee that a certain problem never arises is to guarantee that anything that could be contributory is not allowed to happen.

So, "What does this have to do with me?" you ask. A brief example being better than a long explanation, here is an actual exchange that happened at a DAMA meeting recently. The chapter President called for suggestions for small-group discussion following the main presentation. One of the suggestions was, "Why do we keep making the same mistakes?" The group noted that

  1. it may not be possible to answer the question and
  2. an answer might well be useless in avoiding the mistakes.

Discussion proceeded to other topics. A specific question was asked involving response-time performance in a database application. Many possible causes for extended response times were trotted out without shedding any light. Someone asked about the possibility that a single query, executed repeatedly might be the point of "failure" and included the suggestion that the DBA (database administrator) should be able to say whether this was the case or not. The response: "We don't have a DBA. We thought we could get along without that additional expense."

Another example: Hammering a nail to hold together two pieces of wood is a simple and straightforward operation that can be accomplished by almost anyone. A group of eight-year-old boys could hammer many nails in a relatively short time. If the goal is a habitable dwelling, though, or even a serviceable garage or potting shed, no sane person would entrust the job to the boys. Note that the issue isn't motivation or lack thereof and it isn't tools nor skill per se, it's really about basic knowledge concerning the desired result.

Some theoretical knowledge concerning material characteristics and structural formats is required to deliver a result that will be in service for more than a few minutes.

In the information systems world, the analogy is much more apt than we might be comfortable in admitting. It isn't possible to be involved in a database design discussion (or even a data modeling one) without the term "denormalization" popping up. I'm going to use denormalization as a placeholder for a host of other bits of conventional wisdom in the discussion that follows. Relational data design is an application of set theory which, in turn is a branch of mathematics. Normalization rules (or forms) were developed to ensure that set operations would be applicable and would produce the expected result when applied to a database. Denormalization means avoiding the application of normalization; in practice, it is rarely an activity. There is no methodology for denormalization.

My advice to the supervisor, manager, project manager who is told that the database will be (or has been) denormalized is to ask to see the normalized design and to ask questions about how far the normalization was taken. My guess is that you will get a lot of verbal tap dancing and arm waving. Ask that normalization be explained to you at least through the third normal form. Remember that you will never be able to utilize the full power of your relational database engine using set operations on a non-normalized database. You will always need programming to get at the data.

I have participated in many conversations in which people stated unequivocally their feelings about denormalization and normalization and could not articulate any of the normal forms. The same holds true for statements about standards, methods, tools... It is apparently necessary today to have a strong opinion about any topic that arises. I haven't heard, "I don't know enough about that to have an opinion yet." in several years.

George Santayana may be the first to say that those who cannot remember history are doomed to repeat it. He certainly won't be the last. Data and information design theory has not evolved much in the past 20 years nor have software engineering or project management. Despite that, every day there are dozens of exciting press releases trumpeting the newest (always trademarked) approach to data management, project management, system development...

In the words of the Wizard, "Don't look behind the curtain!" The same difficult, complex work must be done today as twenty (or 30 or 40) years ago. We do have better ways today of dealing with simple repetitive tasks but all that really means is that what is left is more difficult, more complex, more rigorous. This is no place for amateurs nor lone rangers. Every system will be the result of a team of experts working for a common goal and relying on one another completely for their individual expertise. The manager or project manager had better be able to recognize when tap dancing, arm waving and smoke emission is taking place.

Or maybe all you want is cheap and/or quick.

Tuesday, August 18, 2009

The Control Myth

Control is a very much misunderstood concept. The bottom line is this:
Self control is a good thing, an essential thing. Attempts to control others are doomed and will be harmful to all concerned.

This is difficult to write and it is difficult to publish in a public forum. I know it will be resisted and may stimulate reactions from others that will not be beneficial to me. I have made a conscious decision (self-control) and hereby renounce any expectation with respect to responses, reactions, and results.

I retain my hope that the statement may create thought processes that lead others to alter the ways in which they interract with their world.

The conscious reader may note that my statements regarding control are framed relative to people. Control of inanimate objects and such abstraction as process is not only good, it is mandatory. Please note, however, that control of (for example) a process is not the same as control of the people involved in it.

You may have heard of Total Quality Control (TQC) or Statistical Process Control (SPC) and the need to have and use controlled process in order to insure high quality production. A study of these methods has led me to a new understanding of control.

When we think of control we typically associate notions of power. I control something when I can make it bend to my will. A manager is said to control an organization or function. A driver may be fined for failure to have control over their vehicle. This is one of the reasons why TQC and SPC have such a diffcult time gaining traction in the business world in the U.S. (the environment with which I am most familiar). The control that is the core of Quality methods has nothing whatsoever to do with my or anyone else's will. If anything we need to look at it from the opposite direction.

A process is either functioning within understood parameters (in control) or it is not. If it is not, we say it is out of control. What we mean is that we have just discovered that we don't understand the parameters as well as we thought we did. Now we can assert our will to change the process so that the new (improved) understanding becomes part of it. It is potentially life changing to realize that the process is literally in control--that it, is has the control. The evidence is that it produces what it produces. If we desire to change what is produced, we must listen to the process, understand its needs, and give it what it needs.

To take a giant step that may require backfill later, if I have a need to control something, the variability or consistency of output for example, I must give up control to the process. If I have a need to control the people, I must give control to them as keepers of the process. They have a far better ear for what the process is asking for and can give it what it needs. When they can't give it what it needs, they will come to me and tell me what I must do.

To have control, I must give up control.

Life is a process.

Sunday, August 9, 2009

Enforcement and Accountability

Recently I responded to a discussion question on a LinkedIn group forum. The question dealt with how to enforce standards in a data management and stewardship scenario. The other responses mentioned the use of various committees and steering groups as well as management partners for enforcement of standards. One response suggested that a lot of messy people problems could be avoided if automated tools were used to find areas of non-compliance.

I can't help but think that we, as a society, must be nearing the pinnacle (or the pit) of buck passing. When I as an individual choose to ignore an incident in which an action by someone else either ignores the general good or threatens the welfare of all, I am turning my back on accountability and passing the buck to "someone" else.

There have been many instances in which a malefactor, caught in the act, has told me, "What's it to you? What do you care?" If I point out that the action was, for example, in violation of published standards, I might hear, "Nobody follows that. I didn't know it existed until you showed it to me."

The point is that it takes a lot of will in the face of widespread apathy to be accountable for not only following standards, but insisting that others follow them. A study, which I am unable to cite, showed that the rate of deterioration in a neighborhood increases when individual incidents are ignored. For example, a window broken by vandals goes unrepaired or "tagging" of a wall is not erased. Ignoring an incident encourages similar incidents and then worse ones.

I realize I am comparing failure to follow standards with vandalism and I do this with intent. If we assume the existence of a set of standards, they must have a purpose. Normally, the purpose is related to quality. Every manager wants their organization to run like a "well-oiled machine." When an organization does runs this way, we say it is a quality organization. Ignoring a standard is like dropping a grain of sand into the machine. Everything may proceed with one grain, but as one grain encourages two and then three, ..., eventually the machine will break down.

Bottom line: if you're looking for someone else to enforce standards, you're looking in the wrong place. It's up to me and it's up to you. If it's a bad standard, then it's up to me and you to get it fixed. Without personal, individual accountability, you will never get adherence. Enforcement is an empty concept even outside the workplace. People will not endure coercion for long.

Thursday, July 30, 2009

Managing Technology and People

The thing about management is that it works great for money, time or countable things, but management of complex, uncountable things is at best a dream. You say "people are countable" and I'll agree, but only to the extent that the management is about bodies (head count).

Technology and its applications are so complex as to be unmanageable. Edsger Dijkstra spent much of his career trying to get that message across to the business world and the budding computer industry. Today, he is remembered more for optimized search algorithms. If that isn't the industry in a nutshell...

Many years ago I tried to tell people that technology must be controlled, not managed. My slogan was "control technology or it will control you." We are so in love with the notion of management and have so much antipathy for control, that this message, like Dijkstra's, falls on deaf ears. I am not, in any way, comparing myself to Dijkstra. I am comparing his "audience" to mine. Today, the tag line on my web site ( is "Leadership for change, Management for effectiveness, Governance for stability." The three are not mutually exclusive but the probability of finding all three in one person is quite small.

Control is not a bad thing--it is an essential thing. Automobile travel without control would involve so much risk that no reasonable person would attempt it. The control starts with the design and production of the machinery itself. An automobile is designed to be controlled. It is also designed to function within a larger system of controls. Awareness of this allows the designers to prioritize their efforts and focus on differentiators suggested by the system rather than on mere "performance" factors. For example, it would be a waste of time designing a vehicle for mass production that could negotiate a 90-degree turn at 80 mph. The system of controls insures that this level of performance is unnecessary.

Use of technology without controls is also fraught with risk. We require control over the design and production of technology to insure that the product is useful and usable within the larger control framework that is our business context. Control over the use of technology is needed to insure reliability and safety for all just as our traffic laws and their enforcement produce a sense of safety and predictability for those of us on public roads.

Because no one likes the idea of control, we are calling this "governance" but make no mistake, governance must be about controls or any effort is wasted. People want and need consistency. Consistency produces contentment and the role of government, according to SunTzu (The Art of War) is a contented populace.

So, if by management, you mean counting (or accounting), you won't have success applying it to personalities or to technology. If, by management, you mean coercion, you can, for a brief time, deliver the appearance of consistency and contentment with personalities or technology, but you will only be masking a growing problem. If, by management, you mean a system of controls (governance) that produces consistency, predictability and reduced risk, only then will you be able to say that your technology (and your people) are being managed.

Fortunately, the controls necessary for effective implementation and application of technology are well understood (if largely ignored). The Software Engineering Institute's CMMI and ITIL are but two specifications for a system of controls. These are thorough and consistent and understanding them will enable you to create a system tailored to your organizational needs.
The future starts when "control" is accepted and welcomed.

Tuesday, July 21, 2009

Healthcare and Health Care

I have to revisit this subject in light of recent news and developments. It pains me to see the confusion that has caused the pollsters and pundits to be able to take shots at something that everyone wants.

There is concern about cost that is well founded. The problem is that costs come in a variety of disguises. Now discussions of cost have assumed inordinate importance in the questions of access. Once again, healthcare is about cost and health care is about access. These are two distinct issues.

There is virtually no one who advocates the denial of medical care on the basis of inability to pay. The entire health insurance industry emerged in response to a rise in costs driven by improvements in the science and technology of medicine. These improvement demanded better education for medical practitioners (added cost) and the technology has become more complex resulting in increased cost for both development and support.

While all this was happening, we became a nation of sedentary, over-eating, narcissists who believe in the idea that modern medicine can fix whatever we do to ourselves and make us all(well, me anyway) into beautiful people. We have learned to game the system to get the plastic surgeries we want and the pills we want and the therapies we read about.

Insurance coverages have been broadened continually in response to forces too numerous to mention with the result that more premium dollars go out requiring more premium dollars to come in. The insurance companies have developed bureaucratic defenses, requiring second opinions, demanding justification based on diagnostic testing, even making a practice of denial of the initial claim to filter out those who aren't really serious. The additional staffing and data handling is paid for by premium increases and forced out to the medical providers who increase their charges. Rising costs are everywhere and no one can join the debate with clean hands. Everyone wants someone else to absorb the costs.

In the midst of all of this, we sometimes lose sight of those who simply stay away from health care because they don't have enough money to pay for the other, even more basic necessities. We always lose sight of those who try to take care of themselves by buying health insurance, which they can afford only by accepting caps and deductibles or limitations on coverages. These people looks good in the statistics but rarely show up in the doctor's office because paying the premiums has put them into the same category as the uninsured in that they have no financial resources left over to pay for the visit. Further, they now have to live within the insurance bureaucracy that demands diagnostic justification, turning what might have been a $100 office visit into a $300 one.

Access to health care for everyone should be the sole topic in Washington. Access has relatively simple solutions. Let's solve that problem first, and, by the way, we have already agreed that ability to pay will not limit access.

Costs are a completely different issue and one that will require all interested parties to make substantial changes in thinking, planning and delivery.

It is deplorable (to use a word employed by a past President in a slightly different context) that we continue to allow doctors, administrators, insurance CEOs, technology vendors, pharma, and politicians to continue to point fingers at each other while the full cost of inadequate health care is borne by me and you, the patient/consumer.

Wednesday, July 15, 2009

Haves and Have Nots

When I speak of have here, it should be clear that I'm referring to resources. Less clear but no less important things to have include:
  • need (acknowledged)
  • commitment
  • known cause of pain

First and foremost is availability of resources to be applied to making improvements. Data and information quality diseases have much in common with human diseases in terms of diagnosis and treatment. There is much discussion today concerning the state of health care in the U.S. The discussion focuses not on diagnosis or treatment--those aspects are well understood (if imperfectly practiced)--but on paying for the diagnosis and treatment.

It seems that financial resources or the lack of financial resources is the single most important determinant of physiological well being. If we examine the whys behind this, we soon see that expectations have much to do with it. The person without financial resources learns to expect that some problems will be chronic and learns to live with them, perhaps at a lower level of function. The financially well-off person learns to expect that every problem has a cause and a cure and that time and money will produce the expected well-being.

Neither is absolutely correct and both sets of expectations produce advantages as well as disadvantages.

We can apply the lessons of health expectations to data quality. Larger or wealthier companies expect that they will be able to attack a quality issue with sufficient resources to conquer it. Smaller or less well-off organizations will not feel able to dedicate one or more people to the issue and will elect to "do the best they can" (see previous post). Small business leaders will see that everyone must be involved in the solution for it to work and that alone will cause them to turn away from a frontal attack and "make do." Large business leaders may believe that the right manager or leader with sufficient resources can bring it off.

Again, neither is absolutely correct.

A person or an organization resigned to living with pain is always going to find it difficult or impossible to improve while a person or organization immersed in full scale battle with the problem may well miss opportunities for improvement.

As it turns out, a "data quality" campaign is like a campaign against bacteria--almost meaningless. Because the scope and scale of the campaign preclude considerations of nuance, we find that we make enemies from within the ranks and everything degenerates until nothing is happening. We can make progress against a specific bacterium or a specific quality issue but we soon realize that we can't hold those gains without creating a framework within which we can establish trust, confidence and consistency. That framework has come to be called data governance. In the case of physiological disease, the framework is Medicine.

Whether you're a have or a have not, the resource issue turns out to be far less important than we might have thought. Consider expectations first.

  • Can we live with or adapt to the pain?
  • Have we already adapted? How?
  • What limitations are imposed by the adaptation?
  • We can choose to treat symptoms, cure the disease, and prevent the disease. Which is within our reach? What can we do? What should we do?

In most cases, the best choice is to treat symptoms while making lifestyle changes to prevent the disease. Sometimes we have to cure the current disease or we die before we can implement the lifestyle changes. The point is that we always have options. A specific option must consider the past, present and future. A combination of options may produce the best result. Last but not least, have and have not is not really about resources but is about expectations. Commitment is often born of desperation when we realize that we just can't tolerate the future implied by our current expectations. Now we're really ready to do something meaningful.

Thursday, July 9, 2009

Can and Should

Can and Should are in constant tension. They both imply something that has not yet happened--in other words, they both are in the future. So here's the key question:

Do you want your future to be composed of cans or do you want a future of shoulds?

Should is closely related to could.

If you could do what you should do, would you do it? If you should and could but don't, what kind of future do you have before you?

Is your past characterized by "might have", "could have", "would have", "should have", or as my father was fond of saying, "mighta, woulda, coulda, shoulda?"

What's the difference between could and can? It might be knowledge or it might simply be practice. For many people, the biggest difference is the realization that there is something beyond "I can." Parents fill this role as do teachers, mentors and good friends. The process of revealing the new world of could is known as coaching.

What we should do is a function of goals, history and current context. Most of us get paid to know what what should be done. Most of us also take the easy way out and do what we can rather than what we could or should. In fact, "Do what you can," has become a universally accepted surrender. When the boss says it, it means that

  1. they don't know what should be done
  2. they don't know what could be done
  3. they don't want to be bothered with knocking down roadblocks
  4. they don't really care about the outcome

When I say it ("I did what I could.") it means

  1. I know what should have been done
  2. I know that I could have done more
  3. I told them but they wouldn't listen
  4. I was not committed to a quality result

We nearly always allow ourselves to choose the familiar path. When faced with a choice between can and could, we choose to do what we have done in the past--can.

We cannot get the data quality we need unless we have the governance we need and we can have neither if we continue to do as we've always done. This is macro as well as micro advice. Governance is not committees and steering groups, though it may have need of such. Data quality is not one definition, though that may be helpful. Both are about contextual consistency and predictability. This goal could and should be achieved in whatever ways are appropriate to the context within which the consistency is desired.

Consistency is a product of process and the foundation of improvement. Once the process produces consistent output, you have freedom to classify and categorize its output in whatever ways are suitable to its customers. We are currently engaged in trying to classify, warehouse and use inconsistent products created by inconsistent processes.

What could we do? What should we do?

Friday, July 3, 2009

We The People

We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution...
Article 1 of this constitution describes a representative form of governance, recognizing that the needs for deliberation and timely decision making can best be met in this way. This was particularly true in a time when travel was by foot or by horse (or other animal propulsion) or by water propelled wither by wind, oar or paddle.
Two thoughts come to my mind:
  1. What might this article say if written today?
  2. There has been no need to modify the principles set forth during the ensuing 222 years.

All of this leads to a third thought. If the goals of corporate governance are substantially the same

  • more perfect Union--Every CEO wants the company to operate as a unit, with a single purpose
  • establish Justice--A sense of justice is a prerequisite for people to focus on their duties and responsibilities.
  • insure domestic Tranquility--Inter-personal and inter-organizational dissension is a primary cause of lost productivity.
  • provide for the common defence--The company must defend its position in the marketplace and each employee is critical to that defense.
  • promote the general Welfare--This goes hand-in-hand with justice. It's human nature to want things to be better.
  • secure the Blessings of Liberty--Personal liberty is always subject to the other goals.

then maybe we ought to consider whether the method should be the same.

It's hard for me to consider data governance (which is where I'm coming from) in a vacuum. The goals of data governance are substantially the same goals outlined above. Defense is about defending the integrity of the data resource. Union is about consistency. Justice and welfare is about everyone living by the same rules (which produces consistency).

I don't want to make data governance sound so impossibly complex that we throw up our hands in surrender. The message I'm transmitting is that we have models to use. We do not have to reinvent governance.

One of the difficulties in any governance model is to come up with a definition or picture of "the governed". We go through life happily assuming that everyone else is "just like me" in terms of their wants and needs. Mostly that works, but every now and then, we run into someone who isn't "just like me." When that happens we have two choices. Either we try to make the other person just like me or we adapt our view of "me" so that it includes some new parameters. In corporate life, it is exceeding dangerous to assume that anyone in a role different from ours is "just like me."

Even if we restrict ourselves to data governance, we find that we have to include as "governed" many who are filling different corporate roles and are definitely not "like" us. Again, I go back to the American Colonies in the mid eighteenth century. Imposing or trying to impose a set of rules on people whose life and needs I don't understand is destined for failure. The secondary message is: either include everyone in designing the rules or (poor second choice) understand the needs of the others before designing the rules.

Everything I see and hear about data governance is from the point of view of the person whose role is management of the data resource. There isn't a single person in the marketing department who would ever conceive of the need for data governance. Of course, we can spend time in learning to talk the marketing language and becoming familiar with marketing problems, then we can show them that some kind of governance is needed and they will agree. They might even agree to invest some time on a committee. Eventually, though, they're going to wonder if this is a good use of their most precious resource--time.

Making laws (standards) is a messy process. Much of the data governance effort is about the process--identifying stakeholders, building consensus, the political side of things, while the standards and processes become a very small box on a big diagram. My thought is that we don't even know the stakeholders until we understand the processes. The political side is essential, but there is a lot of good we could be doing if we would focus on the processes and standards.

I keep saying this because, while there may be similarity in the way two corporations handle governance, I have serious doubts whether it will ever be possible to export one company's solution to others. The political implications of forcing an outsider's will on a population would cause "failed" to be stamped on the effort nearly immediately.

Bottom line: You're on a burning platform. Don't wait for someone to save you. What do you have? What can you do? Do it!

Thursday, July 2, 2009

Winning the World Series

How do you win the World Series? Implementing good [data] governance is a lot like winning the championship, whether World Series, Super Bowl, Stanley Cup, US Open... It's a big goal accomplished through thousands of smaller ones. It's also similar to achieving Level 5 on the CMMI.
Let's take a look at how to win the World Series and see if we can learn anything about how to implement governance.
The first thing to recognize is that it takes an entire season--it isn't done in only a couple of weeks in October.
It takes the cooperative efforts of an entire organization.
Management must find and hire the right set of talents and abilities.
Coaches must turn the collection of talents and abilities into a team.
Each person must have the desire to excel as a part of a team.
Each person must come to share the vision of Winning The World Series.
Leadership must emerge to keep the vision in front of everyone.
We must win today's game (over and over again).
I must become a baserunner (if a batter) or keep the batter from becoming a baserunner (if in the field). I recognize that I won't succeed all the time but that doesn't keep me from wanting to succeed every time. Winning today's game means winning more of these smaller contests than I lose.
In order to win the small contests, I am prepared. I practice, I consult coaches, I talk with my teammates. I cultivate the knowledge as well as the abilities required.
I choose equipment that fits my needs.
I learn to win the contests in my own environment and in foreign environments.
I cultivate personal and team consistency.
When all of these things are done consistently and well, we find ourselves with at least the opportunity to win the World Series when October finally arrives.
What jumps out at me in all of this is the need for planning, preparation, patience, desire and commitment. I'm sure that no one out there believes that a governance implementation can be launched and completed in a few weeks. How long do you think it should take? Months? Years? Decades? Since their is no finite season or schedule to constrain us, maybe the best answer is that it will take as long as it takes.
That said, it seems incumbent on us to decide how we'll know when we have completed the task we have set for ourselves. I realize this seems self-evident and trivial but as I visit with people and groups I have developed the impression that the stable state is still undefined. What that means is that we are eternally implementing when we should be improving.
In the absence of another definition of the stable state, I have offered two principles for that state:
  1. No [data] pollution and
  2. No nasty surprises

Since these represent a whole series of contests, each of which we are committed to winning, while understanding that we won't win them all, another important property of the stable state is that it embody learning and self-modification (improvement). When we have created the property and the principles, we will have "won the world series". The next step is to understand the contests that make up "today's game" and equip ourselves physically, mentally and emotionally to win those contests.

Today's problem is that we are losing contests that we don't even know we're involved in. There's an old poker adage that says "If you look around the table and don't recognize the sucker--it's you." In [data] governance terms, if we look around and don't see the loser--it's us.

Saturday, June 27, 2009

Feng Shui and Data Governance

I don't know if feng shui is classified as a science, a craft, an art or if it is part of a spiritual discipline, but whatever it is those in the midst of setting up data governance may benefit form some of its principles.

Wikipedia ( provides a nice discussion of the history and principles of feng shui. If we think of data and information as the qi (ch'i) or life force of a business, we can see parallels. We want to encourage the life force to flow through our "dwelling" (the business). We want to hold the good qi and and allow the bad qi to pass through without being retained.

Feng shui begins by studying the physical geography of the dwelling. If the structure is not yet built, the site or sites are examined so that it may be built to take advantage of the natural path of qi. If the structure is already built, everything about its natural geography is taken into account before any suggestions concerning arrangement of furnishings are made.

Holding an understanding of feng shui in one's mind while architecting a governance solution might lead to
  • spending more time understanding what currently is before seeking to change it
  • including all necessary elements (5 in feng shui) in the design such as process, meta data, master data (conformed dimensions), data quality, metrics, intelligence, ??
  • acting small but always within a larger vision
  • recognition of yin and yang (actor and receiver) and the need to consider both within all of the elements

We need our business qi, data and information, to flow freely into the business, through every part of the business, and out of the business. We need to discourage bad qi by showing it for what it is and directing it back where it came from. We need to create harmony within and between business units and functions for the well-being of the business as a whole.

I realize that some readers have already dismissed this with a snort and a sneer. I'm not saying anything about the effectiveness of feng shui. What I am saying is that the goals of feng shui should be the goals of data governance and that it is possible to discover some clues as to good approaches by studying something that has been around for more than 5000 years.

Wednesday, June 24, 2009

Governance and Control

There seems to be some confusion in data governance circles concerning the application of governance--how to make it work. I sat through a tutorial at a recent conference in which the expert emphasized the need for authority as the key (or a key) to successful governance.

It was never clear to me what the scope of this authority was to be or how it was to be used. I finally asked the question, "Authority for what?" You may have heard that responsibility without authority is the recipe for stress and burnout. I thought to pursue this line of thinking as a way to discover what was meant by data governance. If I know the nature of the authority, I should be able to deduce the nature of the responsibility. The question never received an answer. What I got was blank looks.

I felt a strong need to get to the bottom of this since the word "enforce" or "enforcement" was also used several times. I was becoming extremely uncomfortable.

Friends, if people do not accept governance and cooperate with it, then the governance model needs to change. We do not need enforcers. We need arbiters, mediators and facilitators. More than anything else we need teachers. I've heard it said that we all do the best we know how and when we know better, we'll do better.

Controls and attempts to control do not work in governance. They only create bottlenecks and delays that encourage people to find other ways. In our local civil government, we call it red tape and bureaucracy. For example, building permits are required for many home improvements. The reasons for this requirement are excellent. The permit and the resulting inspections (audits) protect the current and future homeowner by insuring that the project is safe. In spite of the obvious benefits, many do-it-yourself homeowners avoid the permit process because the process is obscure, the standards must be discovered, it can be inconvenient, it adds to the cost and is known to produce delays. Furthermore, the only way for the scofflaw to be caught is through an inspection and the authority has no reason to inspect other than the permit. Note that contractors licensed by the authority are much more likely to comply.

Contrast this to the governance of traffic on roadways. Standards are clearly displayed, drivers must pass a licensing test demonstrating both physical capacity and knowledge. Law Enforcement (To Serve and Protect) is primarily tasked with monitoring compliance (which their mere presence guarantees). Compliance metrics are gathered via various kinds of technology and governance changes (to speed limits, traffic signals, etc.) are made based on these audits. What if we had a committee at each intersection with the sole authority to direct traffic?

As you can see, governance requires an initial framework (competence, licensure), a coherent set of standards (coherent in the sense of both understandable and integrated), and monitoring/audit capabilities. Anything else is extra and may even get in the way.

The result of good governance is a community that enjoys consistency, predictability and safety and is mostly free from nasty surprises. The authority that is present is passive and present only to deal with issues that don't fit within the governance structure. If authority is needed everywhere, there is no governance anywhere.

Monday, June 22, 2009

Guerrilla Governance

In the March 14, 2009 post "Guerrillas and Governance" I introduced the notion that, because of long-time inattention to the needs of the people/workers on the frontier (organizational boundaries), systems of governance will have been developed there and may have been in use for a long time.

In most cases, this governance will be relatively crude and inadequate. In the modern context, it might be something as simple as "We don't accept those after 2 PM so that we give ourselves time to get them done before 5:00."

What we, as guerrilla leaders should perceive is in two parts:
  1. This group is dealing with a problem and has a "process" in place for doing so.
  2. There is a problem. It is recurring. It has a cost.

If we feel the need to introduce a new level of governance that eliminates the problem rather than dealing with it on a repeating basis, we must take into account both of these parts.

There is a ready-made community here and they have banded together for mutual protection. We dare not dismiss that fact or we will create opposition that will resist us to the bitter end. Until we take the time to make them feel (not just understand) that we really want to help them with their problem--not ours--they will resist all of our efforts.

The dialogue goes something like this:

you: It looks as though you are experiencing problems with [form, file, request...].

they: You wouldn't believe the kinds of /@#*(^ we get. And it's most of the time.

you: So what do you have to do when you get one like that?

they: When that happens, we have to [lists multiple process steps needed to remediate]. That's why we have to have a cut-off at 2:00.

you: So, if I understand this right, you are getting unusable or unacceptable input from [another boundary function]?

they: That's right. They just don't seem to care how much we have to work.

you: What happens when you complain to them?

they: They just say that it's their job to generate [forms, files, requests] and it's our job to process them.

you: I think there's a good chance that we could guarantee that you wouldn't have to do any of those process steps you told me about or, if you did, it would be rare. Would that make your lives easier?

they: Absolutely. How would you do that?

you: First, we should put together a meeting. I've already talked with them and, believe it or not, they are dealing with similar problems and similar frustrations. I think the solution to your problem is the same as the solution to theirs. To make sure we need to meet because there are still a couple of things I need to get clarified. Will you help?

they: Tell me when and where. I can't meet on Tuesdays at all.

And so it begins. You will use their pain to elicit their cooperation. Their cooperation creates a new community. Community action guarantees compliance. A newly empowered community is a breeding ground for improvement of many kinds.

This is guerrilla governance. The only requirement to get started is a goal. You will need to be able to articulate the goal over and over again in many different dialects. In many cases, you will only want to expose the part of your goal that your audience is able to comprehend. Never try hide the fact that there is more. You'll simply answer all questions openly and honestly and never insist that anyone needs to understand your perspective. "We'll improve our understanding as we go." is a good way to postpone dealing with difficult questions until more education has occurred.

Always remember, you can't do this without them. Their commitment is vital. Talk freely to management about progress and remember that management has pain as well. You're a leader.

Thursday, June 11, 2009


It occurs to me that many people probably don't understand what coaching is or how they might benefit. Since I am advertising myself as a data management coach, the first task in marketing myself may be to do some education on what should be expected from a coach and differentiate coaching from consulting.

If your only exposure to coaching is youth activities or watching your favorite team on TV you may have an idea that coaches call the shots, that they direct, and are to be obeyed. Nothing could (or should) be further from the truth. My job as your coach is to understand what your capabilities are (as well as those of your "team") and to use that knowledge to help you find ways of attacking your goals that are likely to lead to success.

You might also have developed the idea that coaches are cheerleaders and that one of their main jobs is motivation through exhortation. Again, not true. While I will be quick to affirm strengths and celebrate success, I will not create unrealistic expectations. A coach's goal is to help you to understand the most effective ways at your disposal for addressing the problems and challenges that will confront you.

A youth soccer example will illustrate. If you are fast and by nature aggressive, you can succeed as a defensive player by attacking the ball and taking it away from your opponent before they have a chance to score. If you are not the fastest player on the field and are a bit passive or hesitant, you can still produce a good result for your team by merely staying between the ball and the goal and delaying your opponent until help arrives or forcing the play out to the edge of the field.

In data management, similar principles can be applied. An aggressive, direct approach may succeed for some while a more calculated and collaborative approach may work better for others. In any case, you will want your coach to be able to help you find the successful path which calls for experience as well as expertise on his part. One of the least appreciated values a coach provides lies not in what you do but in what you DON'T do. Your coach wants you to be successful and will help you avoid situations in which you can't or are unlikely to succeed.

You have knowledge, talent--all the raw materials for success or you wouldn't be where you are. Sometimes what you don't have is time or some specialized expertise and in that case you will want a consultant who can come in and get 'er done. But sometimes this is counterproductive because you won't be able to keep calling the consultant back each time you need a change or repair. If you have some time, a coach may be a better alternative since he will leave you with success strategies and tactics that you can continue to apply.

You want your coach to be at your shoulder, ready to answer your questions but also to be asking you questions continuously to help organize your thought processes. In that sense a coach is more than a teacher and more than a mentor. A teacher will not be responsible for the application of the subject matter. A mentor may be standing by at the end of a phone line. The coach will be there with you.