Data is like Scat

We (data people) like to talk about data as if it were the most important thing in the world when, in reality, it is exactly like turds.  Yes, that's what I said, merde, gavno, scheiss, shit. 

Of course, it could still be the most important thing is the world but whether it is or is not depends on many things, not the least of which is what we're trying to do.  Data is the byproduct of process much as turds are the byproduct of digestion.

For naturalists and scientists castings or anaimal feces provide important information on a wide range of subjects including the animals diet, where it has been, and its general health.  For a tracker, a different set of information is deducible from the spoor.  The tracker tells which species, how recent, whether the animal is moving or is likely to be nearby and other things that may help him earn a bonus.

No one says, "This is quality shit." or "We need a governance program to improve the quality of our shit."  They simply learn what they can from it and move on.  The kind of value available changes over time but even petrified feces have a story to tell.

If we presume to be data experts, maybe we should be focused on extracting value from our data and understanding the kinds of value that can be extracted as well as how to recognize the data that will tell us what we need to know.

Even the most inconsistent set of data has a story to tell and we should listen to that story instead of wailing about the story we wanted to hear.  Because data is the byproduct of process, inconsistent data should tell us that the process that produced it is inconsistent.  If this is the case, how can we expect consistent data unless we can create a consistent process.

The bottom line is that when we focus our efforts on data quality we are misunderstanding the world we live and work in.  We are creating additional and entirely unncessary complexity.  How does this happen?  The cause lies in the storage of data for input to and output from computer systems.  We have allowed ourselves to institutionalize the cart before the horse.  What computer systems require is consistent input.  A system can be designed to deal with quality issues as long as they are predictable.  A system (process) will always produce output of a consistency equivalent to its input.

If we were pursuing consistency instead of quality, our disagreements would be fewer, our stress would be lower and our impact would be greater.  Consistency a readily understood concept while quality will always be the most elusive of quarries.

Language, Information, Data and Quality

Hat in a Puddle

Language is a slippery thing.  For many years I conducted my life in the firm belief that if I were only precise enough in my selection of vocabulary, clear enough in my choice of syntax, I could convey an idea without ambiguity to any audience.

I have frequently been disappointed with the result. There is a force at work that allows people in the audience to navigate their own way through the meaning of my language.
Those in the data and information industry are accustomed to thinking of meaning as the semantic of the data.  Nothing could be further from the truth.

The dictionaries are full of semantics.  We can choose from a rich set of words (semantic tokens) to describe the situation represented in the picture above.  Note that we can change the meaning of the set of semantic tokens by several non-verbal (without words) methods including tone and inflection.  For example "hat in a puddle" has a different meaning than "hat in a puddle" or "hat in a puddle."

When we talk about semantics, we mean the meaning denoted by the words.  Alas, as humans we must also deal with the connoted meaning that each of us associates with the words.  Words and collections of words invoke in us memories, hopes , desires that are not part of the semantic but are part of the meaning.

The situation is very much like the puddle above.  Most people will accept the puddle at "face value" and simply avoid it so as not to get wet or muddy.  Others will make assumptions and develop expectations based on their personal experience with puddles.  Some of these will not change course, especially if their experience and their current situation allows them to expect that they won't get muddy.  Note that a new dimension was just introduced--a temporal dimension that allows us to react differently now than we might have an hour ago.

All Meaning Not Apparent

A man walking along a road saw a hat in a puddle and recognized the hat as one usually worn by his neighbor.  He thought to pick it up and return it.  When he picked up the hat, however, he saw the face of his neighbor.  He asked whether the neighbor needed help.  "I'm all right.  I've got my horse under me," was the reply.

The face value of words (and appearances) is accepted by most people and used to support decisions of all kinds.

Poets understand that meaning is not conveyed by words.  "Wait a second!" you say, "Poetry is composed of words."  We're both correct.  The meaning of a poem (or a story) is created by all the images, memories, hopes, dreams and desires that those words evoke in us.  This is why everyone who makes rhymes is NOT a poet and why everyone who has a command of vocabulary and syntax is NOT an acclaimed author.

This is the world in which we attempt to improve data quality.  While we may aspire to improve the quality of the semantics, it seems clear that we will never influence the quality of meaning.  This is, perhaps, what "fit for intended purpose" tries to convey.  What if the semantic tokens were musical notes instead?  What if they were colors or smells?  Would we be as confident?

What if we ceased our attempts to control the perceptions of an audience and instead created ways for our audience to explore the boundary between semantics and meaning?

How Would Jesus Do Data Governance (Part 3)

Having tried the preaching route, attracting big crowds, name recognition and a following, (are you with me?) Jesus recognized that the results he came to provide were not being realized. People listened, cheered and went home to the same life they were living before. Anyone who has been involved in the data industry for any length of time will immediately recognize this situation.

Data-driven development, Data Administration, Information Engineering, Data Architecture, Data Quality, Data Management, Data Governance... Cheering followed by more of the same.

What now? Even if you are God and are omnipotent, you still have to get humans to change in order to bring them to the new world--one in which they can rely on one another and in which each acts in light of a defining principle and for the greater good. How long have you been a human? I have more than 60 years of experience in being and being with humans. Jesus was obviously smarter than I. He realized after only 30 years that preaching and teaching just wasn't going to work--even though the Ideal was attractive and clearly in the best interests of all. He was getting, "Well, even if..." and "But everyone would have to buy in..." and "That's not how we understand the policies..."

Maybe this is what you're hearing as well. Maybe you are beset on all sides by scribes and pharisees, nit-pickers and policy wonks. Maybe they are constantly trying to trap you in a bit of heresy or a policy violation. Maybe you've thought about giving up because for some reason the people would rather listen to them than to you.

What did Jesus do? He changed his emphasis from preacher/teacher to minister. He modeled the changes he was talking about and he did it consistently with each and every person he met, meeting each one where they were. He showed them that the past was irrelevant and the future was not a given. He gave them the present and they experienced for themselves that their lives were better.

He did not give them rules, instead he gave them hope and someone to come to. He gave them someone who understood them and who showed them how, by subtly changing their perception, they could obtain victory over the troubles that plagued their lives. He never showed them judgment or condemnation. He always cared about their welfare and he planted seeds of change.

He gave himself for the people long before he gave his life for them.

If you believe that better policies, better standards, better rules, better laws will force people to care about data, I'd like to help you and I hope that I may already have helped you by planting this seed. Sitting in an ivory tower and sending out criers to inform the people of the duke's latest whim will never (as in NEVER) be productive. Maybe you'll think about giving up some day and you'll remember this seed.

What Is "Data" Anyway?

If you have any experience with phone support (on either end) you will recognize how easy it is to get deep into a process before realizing that the other person is on a completely different path than you are. RTFM (Read The F---ing Manual) often pops up as the answer to our communication difficulties, but it clearly is not the answer or it would have been universally embraced by now.

I happen to be a proponent of the theory that the answers aren't nearly as important as the questions. As a teacher, I know that learning is happening when the pupil is asking questions--particularly a series of related questions. This has become very important as I have attempted to make headway on [data] governance and [data] quality.

My early attempts assumed that everyone knows what data is--and they do, in the same way that a picture is worth 10,000 words. Each and every person you talk to knows what "data" is and each has a different idea in mind. For most, it's a picture of the last set of values they looked at. This might have been a spreadsheet, a graph, a collection of measurements... The key is that data is a set of values. For some, "data" is a commodity. It is files, stripes on a disk, a percent of capacity, a quantity of bytes measured in "mega-", "tera-" or "peta-". For still others, "data" is represented by a schema, model, definition, or some other abstraction.

Given those varying perceptions or perspectives, is it any wonder that at some point in the quest for "data" anything we find ourselves stuck in the quicksand of confusion. Even when all parties have been saying the same things and have been agreeing on goals, there comes a point when someone will say, "We're not going to do that." or "I don't see why that is necessary." or "But that will change my work flow." This is frequently the point at which everything starts to unravel.

So what I have learned, and what I offer to you now, is that the initial phase of any data initiative must be a carefully constructed education process to insure that the quicksand moment never happens. This must be thought of as risk management. Remember, too, that the really important questions (and answers) initially are not the ones you're hearing. All of the different constituencies are going to be much more comfortable exchanging information (or misinformation) within the tribal group than with "outsiders."

The best way to head off this risk is to carefully choose allies from each constituent tribe and use informal conversation about their pain points and the ways that data figures in the relief of that pain. You will be setting these people up to be the "experts" within their respective tribes. Part of this will be coaching them in how to respond to questions and discussion in which they don't feel themselves to be on firm ground. They need ways to postpone a response until they've had a chance to confer with other experts. This is easy to do by setting up a collaborative model.

Rule one of this model is that I never answer for someone else. Everyone can understand that a situation involves yet another perspective and that it is necessary to involve someone from that tribe in order to get a complete answer. The most common danger here is "We don't have time for that." Everyone must understand that this is an absolute red flag event. It signals that we still have not achieved a universal understanding of objective.

When a red flag event happens, it isn't the same as finding ourselves up to our necks in quicksand. It just means that we need to engage in some risk mitigation. It's a sign saying "Quicksand ahead." We will need to bring this person into the fold--usually through informal and non-threatening discussion with at least one peer or trusted expert.

Your role, should you choose to accept it, is to be a non-judgmental, constant, committed, and helpful presence that can be relied upon to be a neutral mediator and facilitator who feels like a friend in any need. Your motives must be above reproach. You cannot count on and should not hope for recognition. All around you will be better off for your presence.

If you are senior in the organization to this person, you should make sure that you are appreciating their contribution but they will appreciate non-public affirmation since putting them in a spotlight may negatively impact their ability to continue to function in the same way.

"What is data anyway?" is a question that requires many answers initially and one answer eventually. Remember, though, that many people are really only interested in what they have to do differently. "Data" may have no meaning whatsoever in their day-to-day responsibilities even though they may be monitoring real-time run charts with instructions to take a specified action when the line goes above or below a certain point. You can't possibly know where to start or where to stop in defining data for them. That's why you need the tribal expert.

Don't seek "important." "Helpful" will take you much farther more quickly.

Introduction

I never had any intention of becoming a blogger--not that there's anything wrong with blogging. It may be too little, too late and I certainly feel like the boy with his finger in the dike, but there needs to be a voice of reason in cyberspace when it comes to the latest HOT topic, Business Intelligence.

I am leveraging more than 20 years of data management experience here, so I'm going to try to avoid the bling and get to the meat. Since so much of the BI (short for business intelligence) stream is bling, frills, bells and whistles, I should be able to keep these relatively short.

I'll inaugurate this spot with the assertion that if there is no meat to what you're seeing--if the information isn't "actionable", then it doesn't matter how it's presented. That means that if you've blown your budget on presentation (ex., dashboard) tools but you've never spent a cent on a data quality assessment and you don't have process consistency or even documented processes, then you won't get intelligence.

That's enough for an intro. More tomorrow.