How Would Jesus do Data Governance?

Recently I've been led to a reassessment of my approach to this whole blogging thing. Originally I had the idea that people would be attracted to a common sense, git'r done view of data governance. This quaint notion was driven by the fact that, although a lot of money was changing hands under the label "data governance" not much was really getting done to realize the vision.

There have been several incidents of late that have honed this approach by grinding away some remaining misconceptions on my part. What is left is the sharp edge--no assumptions, no vain hopes, no vanity, no illusions or delusions--just a cutting edge that can be applied to any knotty issue in the data governance/data quality landscape. Actually this edge can be applied to any knot whatsoever.

First I had to let go of the idea that anyone would listen to and heed an idea just because it's good. We've come to expect marketing glitz. A wise person once said, "All that glitters is not gold." The best ideas are seeds that must be tended over time but which will, in their own time, produce fruit.

Then I had to abandon the notion that people would rally to an idea. As it turns out, people rally to people--to leaders. Here I would ask the reader to consider leaders in general and some specific leaders. What was the basis of their "leadership?" In most cases it was charisma. Too often their followers realized too late the direction in which they were being led. I'm not charismatic.

Finally, I have understood that people will follow reward (most often money) rather than ideas. The great idea of data governance has been crushed, chopped, sliced and diced in search of marketing leverage and greater monetary reward.

All of this honing has left me with a better appreciation of Jesus of Nazareth, known as the Christ. I'm not comparing his mission with data governance--that would just be ridiculous. I do want to look at his methods, however, since many of the problems are strikingly similar.

Stay tuned here for an exploration of "What Would Jesus Do?" applied to data governance.

Chapter One

This will be chapter one of the eventual book. Some things have to laid out very explicitly. No one should ever be able to complain that they didn't get their money's worth or that they thought they were getting something different.

Some basic principles that will govern everything else that is said:

• Data is part of language
• All communication about data is, itself, data
• Language consists of denoted meaning (denotation) and connoted meaning (connotation) and on top of these is layered implication and inference which involve human perspective
• Nothing about language guarantees communication
• Communication requires a minimum of two entities from the following set [human, machine, logical construct (e.g., software)]

If we are to get a grasp on data quality we have the best chance of success if we restrict our discussion to ONLY that data that is part of communication between machines or between machine and logical construct. Of course this is neither exciting nor even very useful. We have many specifications formats (ethernet, etc.) that guarantee that communication on some level will take place between machines and logicals. ASCII or EBCDIC are the most basic of communication specification. It doesn't take very long for the alert observer to notice that unless a human is involved somewhere in the process, it doesn't really matter what the communication is.

"Matter" implies human involvement or at least we can infer human involvement from a statement that something does or does not matter. Matter is a value judgment couched in an emotional context. It's only when we start to peel back the layers, asking why or in what sense something matters that we begin to get to the idea of quality. Our exploration, then, will follow the trail of what matters.

As we follow this trail we're going to encounter the idea that what matters is, in many ways, distinct to the judge. What matters to the reader of a graphical novel (formerly comic book) may not be the same things that matter to a reader of War and Peace or a viewer of The Mona Lisa. What matters to someone watching Wile E. Coyote fail in yet another attempt at catching the Road Runner is not the same as what matters to someone watching Being John Malkovitch or Inception.

How then do we determine whose perspective to assume? Whose view matters?

The answer of course is that, where communication is concerned, everyone's perspective matters. It would be a great feat of communication if we could present a context-free (perspective-free) discussion of data quality. In fact, it would be such a feat that we're not likely to ever see it and it certainly won't happen here. Our intent is to zero in (or home in but NOT hone in) on a very small number of perspectives to see what matters to them and then step back to see if there are any common themes that can be exploited. If we are successfull in that, we may have created a springboard for the one who comes after.

In the next chapter we will nominate some key perspectives and begin to investigate what matters to them.

DQ: More Than Meets The Eye

As we move forward toward a view of data quality that allows us to create and use a language specific to DQ issues, descriptions and solutions, let’s take a minute here to examine the behavior of data.

Certainly, one of the attributes of quality data is that it is well-behaved. In other words it consistently delivers value according to principles that are applicable because of its type, domain, range, relationships, maturity, purpose(s)…

It is useful at this point to differentiate between static and dynamic properties of data. Any DQL (data quality language) that we might define should work well where static properties are concerned. When we begin to consider dynamic properties, the task becomes much more complex. The greater the number of dynamic properties, the greater will be the complexity.

Our chances of designing a DQL will be significantly greater if we can restrict ourselves to static properties only. Before we can do that, we have to understand the dynamic properties and assess their relative importance. Can we carve them out of the discussion? Will excluding them compromise our DQL’s capabilities?

Looking back at the list in paragraph 2, the first three properties might be thought of as static. These are the focus of our modeling efforts or, if we only pretend to do modeling, of our programming efforts. There is a tangent here that we’ll resist for now, but at some point we have to come back to it. The question of how data is initially defined is huge and the effect of initial definition on the lifetime of a datum and in particular on its quality is not to be underestimated.

For now, though, we’ll put that on the back burner. We expect the individual pieces of data to possess a definition (usually called a description), and our DBMS requires that we say what kind of data it is. Is it variable length text strings, a specified number of characters, integer, floating point, money, date/time, etc. It is surprising how many data are defined to the DBMS as varchar. It shouldn’t be surprising since all of our modeling tools allow us to set a default type and the default for the default is always varchar(n). This is popular because it guarantees that any value supplied will be accepted. Oops, another tangent almost sucked us in.

The final three items in the list are dynamic in the sense that their values can and will change, sometimes rapidly and usually unexpectedly. Let’s take the last first. Purpose, as “fit for…,” will change whenever we’re not paying attention. We hope that our stewards will be on top of this but pragmatically (everyone likes pragmatism), they may be too close to the business itself so that changing business needs or drivers loom so large that defined purpose fades to insignificance.

Maturity is also dynamic. We expect maturity to change over time. When we think of data maturity (if we do) we include stability (of all the other properties), quality metrics that have flattened out, recognition within the enterprise and probably several other aspects.

Finally, we have to face relationships. We’re not very good at relationship management. Some of us wouldn’t recognize a relationship if it sent us a valentine. Others pile all sorts of unwarranted expectations on top of our relationships and then wonder where has the quality gone.

It all starts in the modeling phase. Chen, when he invented a graphical notation for describing data, gave equal weight to entities and relationships. Both had a two dimensional symbol and the opportunity to possess attributes. For many reasons, not least perhaps that tool developers didn’t grasp the importance of relationship, “data modeling” tools eventually turned a multi-dimension, real thing into a single one-dimensional line that is only present all as a clue to the schema generator to copy the identifier from one of the linked entities into the attribute list of the other and label it as a foreign key so that the database engine can build an index.

Although I find examples are often counter-productive in the discussion of data quality, one example may illustrate the role of relationship in completing the semantic of a data set. PATIENT is such a common entity in the health care marketplace that no one even bothers to define it. It is a set of “demographics” by which we mean the attributes and it has relationship with PHYSICIAN or PROVIDER. It probably also has relationship with Visit or Admission, Order, Procedure, Prescription, Specimen and other entities of specific interest to the enterprise such as EDUCATION_SESSION, CLAIM…

It deosn’t take long to figure out that the relationship between patient and physician is more complex than can be accommodated by a single foreign key. A physician can “see” a patient, refer a patient, treat a patient, consult (with) a patient, admit a patient…the list goes on and on. Each of these relationships has real meaning or semantic value and may even be regulated by an outside body. Typically, these are implemented by a single foreign key attribute for each.

Now, imagine a situation in which an in-utero procedure is scheduled on a fetus. You may be aware that transfusions, heart valve repair and a host of other medical procedures are actually performed on the fetus while it is still within the mother’s womb. So, who is a patient? If the facility also terminates pregnancies for any reason you can see the conundrum. Medicine doesn’t allow for terminating the life of a patient (Dr. Kevorkian excepted). At the same time, we would like to sometime treat the fetus as a patient, perhaps for reasons of safety. We also experience the lack of values for attributes that we may have viewed as mandatory, e.g., DOB, SSN.

It is only when we explicitly talk about relationships that these issues emerge. Relationships cast light on the entity from all angles.

Relationships also represent the business processes that inform the purpose of the data. Often, undocumented meaning gets attached to data. Two analysts will get together and agree that for the purpose of this analytic, this combination of attribute values will be included (or excluded). For a given ETL job, we decide that an attribute value that isn’t on the approved list will be replaced with “&”. The adjustments to business processes are constant and usually undocumented and unnoticed. Until we can point to a documented process/relationship, we have no way of capturing and dealing with changes.

What’s the difference between an association and a relationship? Somewhere in there we’ll find clues about dynamic quality properties. One thing leaps out as a property of quality and a property of relationship—expectation. When we claim that something has quality, we establish an environment in which it is permitted to have certain kinds of expectations. The same is true of relationship. When two parties or entities enter into relationship they agree as to the expectations they will have of each other.

In our quest to define quality for data, we will be forced to document expectations and to monitor accountability with respect to those expectations.

Data, Governance and Data Governance

There have been some great discussion threads on the IAIDQ LinkedIn group recently. One thread that attracted a lot of attention started with a question from a PhD candidate in Data Quality. It simply asked whether there is an accepted definition of data quality. 200 replies later, most people would say, "No." More recently a thread began by bemoaning the fact that there is no accepted definition of Data Governance. A lively discussion followed that continues even now. Yet another refers to an article on five reasons to cleanse downstream instead of preventing upstream.

I am about to shed some light, however feeble, on the subject. Allow me to start by admitting that I am a person who likes to do the analysis necessary to solve a problem. Though my patience is improving with practice, those who know me will back me up when I say that I want to solve a problem ONCE.

Previous posts here have explained the abstract nature of data and all that implies in terms of getting people on board when it comes to doing something about quality. People will listen to or read a horror story about some preventable data issue that cost Company ABC $umpteen million. They will nod sagely and say something like, "They should have seen that coming." They are simply unable to see that their own company is engaged in the exact same practices and completely at risk for the $umpteen million.

Friends, it isn't just our favorite whipping boys, Management. There is no more recognition within IT than there is in the boardroom. Our boxes and wires friends think of data in terms of DASD and Raid configurations or bandwidth and throughput. Our developer pals don't really think of data at all except as the fuel that activates their code. Architects appear to be concerned with the storage and throughput views overlaid with an access management filter. They seem more concerned with making developers and DBAs happy than with the quality of the asset.

Enter Data Governance, which in most instances wants to be about definitions, rules and "enforcement." Often Data Governance tries to heap another thick layer, called meta data, on top of all the data that is already being mismanaged in the organization. It's often the case that Data Governance fails to practice what they preach.

Here's the revelation: Data Governance isn't about data. Data Governance is about process. It is the means to the Data Quality end. I have already said that Data Governance is that part of corporate governance that is dedicated to stewarding the corporation's data asset. It is exactly analogous to the role of Finance/Accounting with respect to the capital asset. Unfortunately, Finance has two things going for it that Data Governance doesn't have: GAAP and audits.

Generally Accepted Accounting Practice is a set of guidelines for money management processes that are accepted as the name implies and USED nationally and internationally. The use of these practices insures that processes will be auditable. The audit process verifies that GAAP was used and if there were exceptions, that they were clearly noted with enough information to allow the results to be brought back into alignment with GAAP. The underlying theme is that if the processes were sound then the result is believable.

Imagine if every company of any size whatsoever were able to devise and use its own bookkeeping structure and process. There could never be a stock market. Equity trading would be too risky for anyone and all business would essentially be sole proprietorships. Moreover, there would be no chance of oversight by outside bodies (Government).

This is a picture of the situation with respect to data today. When will it get better? Data Governance has no power to make the situation better. Without an externally defined data management framework and periodic audits by independent auditors, there will be no improvement. In the meantime, if data quality metrics improve, it's only because some particularly strong and charismatic personality is present.

No one questions the need for accounting nor the rigor of accounting procedures. Actually the same can be said for data governance and data management procedures. The difference being that in the case of money, the lack of question results in compliance while in the case of data it results in apathy or confusion.

Does the data world have something like GAAP that could become the necessary process infrastructure to support data management audits? I don't see it. Data is still too personal, too subjective, too misunderstood to attract the attention of researchers. Data management is a black box to virtually everyone and they like it that way.

People prefer to cleanse downstream data because their customers fell their pain being relieved. Happy customers is the goal after all. The bonus is that cleansing provides an unending source of employment for those doing the cleansing. It's win-win! People aren't going to be highly motivated to change a win-win scenario any time soon.

Whose Job Is It, Anyway?

There are several reasons why you are reading this blog post. Leaving out all the self-aggrandizing ones, let's focus on those who actually have the title question in their minds. You may have been drawn here by an interest in BI (business intelligence), the latest name for "reporting." It is likely that you have had some bad experiences involving reports or dashboards, or mash-ups or some other information display/access effort.

You spent what seemed like way too much time getting to an understanding of

• how this this was to be used
• the kind(s) of content that would be useful/acceptable
• how the information should be arranged/displayed

If you are part of a really accomplished organization, you may also have had seemingly endless discussions concerning

• How "bad" data would be recognized
• How "bad" data would be handled
• Remediation or cleansing processes to reduce the incidence of "bad" data

And, finally, if your organization is in the six-sigma population

• What is "bad" or poor quality data?
• Where does it come from?
• What does it cost?
• Where should we devote our efforts?
• What kinds of efforts offer the greatest ROI?

If you follow the various discussion forums concerning data quality, you will find one question popping up with regularity: "Who is responsible for Data Quality?" I asked myself why the question is asked. What prompts the question? It seems not to matter whether the organization has a reputation for quality, nor whether it has a history with data quality, nor whether the questioner is experienced or inexperienced, executive, manager, or front-line production. Why?

Having talked with some of these folks and researched the situations of several others and then simply meditated on this over an extended period of time, I have come up with a few likely scenarios.

1. The questioner knows the answer but either wants validation or a sufficient number of the "right" answer from people who are likely to be respected.
2. The questioner has encountered roadblocks from unexpected directions and is dealing with surprise and disappointment.
3. The questioner is curious about what others are doing.

What I have NOT seen is any evidence that the questioner is sincerely trying to determine how best to attack the problem of poor data quality. It would be easy to assume from these same discussion forums and conversations, that nearly everyone knows what they're doing and has either solved the problem or is well on their way to a solution. When we learn enough about human nature we understand that these people are all feeding their egos (or rather that their egos are feeding themselves since much of this is unconscious) and that they are taking an incidence of limited or small-scale success and projecting it into an eventual enterprise level solution. I have done this myself.

I know this--that data quality, or any other variety of quality for that matter, will not bend to advanced degrees, nor to mastery of data and information design concepts, nor to any product or set of products, nor to any effort by marketers, nor even to the best-designed procedures, methods, governance structures, architectures...

The ONLY way to data quality lies through the hearts and minds of each and every person in your organization. Each of them has the power to subvert any plan, procedure or method; to render ineffective any tool or product; and to humble the greatest of egos.

The bottom line is that it must be everyone's job but of course that isn't a satisfying answer because the very short list of things that everyone believes are important includes things like breathing, eating, elimination of wastes, procreation, maybe community, relationship, acknowledgement... This list of universally agreed-upon important things will never spontaneously include data quality. In fact, if a poll were conducted in the boardroom, it is unlikely that data quality would appear on the list of things important to the company.

Don't misunderstand--the quality, security and accessibility of your data assets is at least as important to the continued health and well-being of your company as that of your capital assets. The problem lies in the fact that data isn't real and tangible. If bad data smelled bad or rusted or developed crumbling holes, or if it resigned and went elsewhere where it was more appreciated or was subjected to audits by outside entities, or showed up on a P&L or balance sheet where it was reviewed by prospective data contributors--THEN it might get some attention.

It is true that anyone can produce an example of data but virtually no one--even those responsible for collecting and storing its instances--will understand that data is something other than what they are holding or pointing to or storing. But I digress.

If we can't accept the answer that data quality is everyone's job then we need to move on to identify the person or corporate function who will be accountable for the quality of the company's data. It's not possible here to put a name to this accountable party. What we can do, though is itemize some of the skills and abilities required to help point the way to your unique name.

First and most important, let's agree that what we are talking about here is cultural change and cultural change, more than any other kind of change, requires leadership. Already we see that a corporate function can never be accountable although you can tape a job title on the person's door when you identify him/her. This leader will be able to move freely across the company and will be able to give everyone the feeling that they have been heard. This DQL (data quality leader) will be conversant with principles and practices of quality improvement. The DQL must be completely comfortable with the nature of data and will not be confused by the display of samples. Attributes of the data asset as a whole will be the focus of all of the DQL's efforts. S/he may well choose to shine the spotlight on a segment of the population and may delegate someone more familiar with that segment to assume the leadership of this effort. That surrogate DQL will also deal only with population attributes (metrics).

The DQL will never fall into the trap of confusing examples with anything else. An example may be representative or it may be an anomaly. Only the population metrics allow us to tell the difference.

Further attempts at guiding your choice may be counter-productive if you begin to feel manipulated or otherwise used.

A final caution concerns those characteristics that will render a person unsuitable. Just as you might wonder about a carpenter who feels compelled to talk interminably about his hammer or his saw, the person who leads with the name of a tool, tool vendor, methodology, author, book, etc., is unlikely to be the one you're looking for. The last thing that the leader/agent of cultural change needs is to divert any attention away from the primary focus. Products and tools may be useful for producing the population metrics discussed above, but beyond that should be well in the background and completely invisible to the majority of those you are attempting to influence.

Those who confuse files or [mega/tera]bytes with data are likewise unsuitable as are those who confuse a spreadsheet, chart or report with data.

I hope I haven't made it seem like an impossibility. Talk with people about this and over time you'll begin to get a feel for what to look for and what to avoid.

Leadership, Management, Governance

Leadership for change

Management for effectiveness

Governance for stability

It is way too easy to become confused where these three functions are concerned. We like to think of ourselves as leaders when what actually consumes our time and attention is hitting deadlines and deliverables. Leaders emerge when change is in the air.

When the marketplace is shifting; when the economy is deteriorating and taking our profitability down with it; when we're no longer able to keep up--that's when leadership is required and leaders step forward. Anyone can steer a straight course through calm seas with good charts. It takes a master to sense the environmental factors, inspire confidence among the crew and make the continuous changes required to keep the ship from breaking up or running aground until things settle down and we get back into familiar waters. We risk disaster when leadership isn't acknowledged and permitted to assume control. When the crisis is over, we often find that the leader has little or no interest in the day-to-day operations of work rosters, schedules, performance reviews... The leader may make a very poor manager.

The manager is the master of routine. She is the one who keeps the machinery humming and the product going out the door. He makes sure that time boxes are hit, that deliverables are delivered and that budgets are created and followed. When the winds of change begin to blow, the manager who fails to recognize the need for leadership or believes in error that he can handle the leadership role can cause massive and sometimes irrecoverable damage before she agrees to [temporarily] relinquish the helm. Exceptional management defines the team, builds the team and keeps the team vision alive. This management is essential.

Governance is ubiquitous and invisible. Governance is observing, analyzing, formalizing, monitoring, measuring, improving. Governance establishes the standards to which managers hold themselves, each other and their teams accountable. It should be readily apparent that governance is every bit as necessary within a high-performance organization as is management. An exceptional governance function is a combination of historian, engineer and seer. A liberal dose of management is required to insure that governance doesn't degenerate into approval by the de facto expert.

The six sigma governance function will have incorporated leadership into the system of standards. A set of standards and standard processes may be of little use to the leader in the midst of the storm but may have helped to prepare that leader to be able to step forward.

Leadership for change, management for effectiveness, governance for stability is the tag line of WhiteLake Data Management. Data Management will be a microcosm of the enterprise as a whole. Every point of view (perspective) within the enterprise will be represented in Data Management. Virtually every process within the enterprise will be examined by Data Management in an attempt to "get it right." The risk associated with unreliable information (data) can only be assessed in light of the process(es)--and the personalities-- involved. Data Management is not governance but must include governance as an essential component. As its name implies, management is its bread and butter. Because it is frequently considered the homely step child, however, the availability of leadership may well be the key to its success.

Data Governance Is...

Recently someone on a LinkedIn discussion forum asked for a definition of data governance because he had yet to find one that was universally accepted. Now he has about 30 definitions and all of them are "personal" in the sense that they apparently work for the person who responded and/or that person's company.

In fact the lessons of the past 10 years or so produce the conviction that data governance is anything, everything and nothing.

It is anything we need it to be that serves our immediate purpose.

It is everything in that it spans all corporate functions in order to produce the needed results.

It is nothing because we always have to define the term whenever it is used and no two definitions are the same.

My own input to the discussion was that

Data governance is that part of corporate governance that is concerned with insuring the integrity of the corporate information resource.

The only useful application for this definition is in establishing a context for any initiatives and establishing responsibility or accountability. This definition cannot be used as a strategy or a vision to drive results. It doesn't suggest any metrics. It doesn't help us to isolate key processes nor does it suggest any standards.

Other definitions you may have seen involve "decision rights". The problem with "decision rights" is that there are always individuals who will pop up once one of these rightful decisions is made and insist that the decision in this particular case was rightfully theirs. Quite often this individual will make a good case and the "rightful" decsion will be overturned--often at considerable cost. When this happens, it calls into question the makeup of the existing decision-making bodies and can cast a long shadow over the entire concept.

My definition contains a problem in that it invokes another poorly defined concept--that of corporate governance. As I have discussed in previous posts, corporate governance in any form that would support the needs of those desperate for data governance is as rare as a polar bear in west Texas. It is the most challenging, most demanding and most thankless job imaginable to create a system of governance in the midst of a feudal culture.

This is not to say that results--even valuable and far-reaching results--can't be obtained. Such results are possible for those who are dedicated, courageous, knowledgeable and visionary. If one can keep the vision of data governance as part of a corporate culture and pursue integrity for all information but do it one relationship, one entity--even one attribute--at a time, then real progress can be made.

I have often wondered why we think we need a definition for data governance when it is so obviously subjective. Of course data integrity and data quality and even data itself are equally subjective. None can be discussed without first offering a definition ("What do you mean by that?") and we don't have definitions that we can quote that are meaningful to "this" audience. In fact, we are given to definitions that are nearly meaningless even to our colleagues and serve only to get us all in the same ballpark.

It is possible to take any result and call it data governance (who could argue?). It is possible to take any corporate initiative and use it to promote data governance. Why don't we simply get busy and spend our time discussing results instead of definitions? Show me the data that demonstrates you have brought processes into control for some subset of your company. Let's talk about which processes are most critical and which represent the greatest opportunity. Let's get moving.