Setting the Stage
In the digital age, data is being generated at an unprecedented scale. Estimates show that in 2025, over 180 zettabytes (1 zettabyte is 1 trillion gigabytes!) of data will be generated worldwide, an increase of over ten times compared to figures in 2015 (Richter). From transaction records and sensor data to social media interactions and academic journals, a tidal wave of information is upon us. Unfortunately, the scale of this data can be more overwhelming than enlightening if it is not dealt with correctly. While more data promises better informed decisions and more powerful artificial intelligence (AI), it can also confuse us, cause inefficiencies, and breed bias. These days, it is imperative not to focus on how much data is flowing around but to rather ask, what data is relevant? More importantly, who and what defines relevance?
We are at a turning point of data, we can either let it control us or take steps to maintain control over it. AI models trained on a plethora of human generated data have the potential to change the way we interpret and act on this data. However, these models are only as good as the information we feed them. This is why it is paramount to understand the quality of data we have at our hands. Clean, structured, and relevant data can provide clarity and insight while disorganized and unfiltered data may be misleading and contribute to bias. As we consider this we should ask, what is the making of a “clean” dataset and how do the AI models that train on this data react? What gets excluded and is it more important than we think?
This digital essay explores the idea of what is relevant in a data-flooded world and how it is shaping the still emerging technology of AI. It looks at this idea technically in regards to the data cleansing techniques employed today, but also through a social lens that analyzes how humans are affected. As we enter an age of information saturation, we need to consider what it takes to build technology that accurately accounts for the complexity and nuance of our world.
The Importance of Relevant Data
The graph below portrays a serious imbalance between usable data and the total amount of data in the “global datasphere”. This sphere, defined by the International Data Corporation (IDC) as the sum of all created, replicated, and captured data in three major areas of digitization. The first area consists of the traditional and cloud data centers that are primarily responsible for storing, managing, and facilitating the use of data, called the “core” of the sphere. The second location is called the “edge” and is made up of enterprise infrastructure like cell towers and branch offices. The third and final component of the sphere are the “endpoints” which consist of electronic devices such as PCs, mobile phones, and all internet of things (IoT) devices. IoT refers to interconnected devices ranging from household appliances to vehicles and encompasses the largest area of consumer interaction with technology.
It turns out that all of this electronic infrastructure leads to an abundance of data collection, so much so that most of it goes unused. Conservative estimates from Seagate show that only about 2% of all of the data within the global datasphere is actually stored or analyzed at all. Most data, especially in the edge region is rather fleeting being “discarded, overwritten, or sensed, but never stored longer than milliseconds”(Reinsel) . One reason for this lack of inventory comes down to limitations of data storage, but another critical reason is dictated by what enterprises deem relevant. Data is relevant when it enhances user’s experiences “they want data in the moment, on the go, and personalized”. This is increasingly making real-time data the most relevant data to companies and consumers alike. If data is not context-aware and customized it is likely to be ignored or ineffective. However, that is not to say data is irrelevant if its immediate need is not apparent. Businesses also want to hold onto data that can be used in the future to enhance their technology and services. At the end of the day, a staggering amount of data goes unanalyzed each year due to what users and business alike deem relevant to their individual experience or long term goals. It is exceptionally difficult to know whether the data that is unusable could be put to use in beneficial ways because this data is often flowing through proprietary channels whose overseers decide whether it is worth analyzing further. Ultimately, data relevance is determined by the users and collectors of it and it has no clear-cut definition.
The table above is a crude representation of the total data generated in the global data sphere versus the amount of data that is usable. The total data generation information is provided by Statista (Richter) which conduct their estimates by pulling data from other sources like Seagate, Ohio State University, Ciena, and Western Digital. Usable data information is provided by Seagate (Reinsel). It is important to note that each of the bars plotted on this graph are represented as numerical estimates based on a variety of industry surveys and data analytics. This graph is not accurate with certainty but provides a striking juxtaposition of metrics that highlights the absurd scale of data generation in the world today and how we are only able to handle a small fraction of it.
How Data is Cleaned
Data cleaning (also known as data cleansing) is the process of identifying, analyzing, and correcting errors that arise in datasets. Imagine you are following a recipe that tells you how to bake a cake and one of the ingredients it calls for is “2 cups of flower”. As a human, you would probably correctly assume the author meant to write “flour” and carry on with your cooking without issue. However, computers are not great at making these seemingly simple decisions on the fly with much accuracy. While there are likely not many computers in charge of following cake recipes, misspellings like these are just one of many issues that contribute to data inaccuracy and dirty datasets. In all datasets, especially large ones, it is reasonable to expect the presence of misspellings, invalid or redundant data, and missing entries. Unfortunately, these issues can be a pain to deal with. The IEEE Computer Society’s Data Engineering Bulletin and Van den Broeck et. al’s policy forum on data cleaning both speak on these issues in the business and medical research worlds, respectively.
In IEEE’s collection of papers, the research tells us that most data is messy due to inconsistencies in data formatting and cross-platform incompatibilities. For example, when a survey asks you for your name you might put it in a different format than other users (i.e. P. W. Schmidt vs. Phineas Schmidt). Someone else may forget to enter their name at all and the data will have a missing component. Organizing data that is built using different formats is a tedious issue which must be addressed before the data can be used for analysis. Furthermore, when merging data from multiple source such as web logs and spreadsheets problems like duplicate entries, inconsistent schema (model) structures, and conflicting units of measurement arise. Data cleaning attempts to resolve these issues by standardizing inputs, detecting and eliminating duplicates, validating data against known rules, and sometimes even predicting missing values. According to Rahm and Do in the IEEE bulletin, this process is an essential prerequisite for any decision-making system or analytical tool, as “garbage in” simply leads to “garbage out.” In short, data cleaning ensures that the data being analyzed is accurate, consistent, and useful (Rahm, et al.).
Van den Broeck et al. emphasize that data cleaning is a critical component of the research process, particularly in clinical epidemiological studies. They propose a structured, three-phase approach that consists of screening, diagnosing, and editing. This methodical process helps identify and correct errors that may have occurred despite careful study design and implementation. The authors argue that data cleaning should not be viewed as a mere technical task but as an integral part of study design, deserving thorough documentation and reporting. They advocate for transparency in describing data-cleaning methods, error types, and the impact of cleaning on study outcomes. This perspective highlights the importance of data cleaning in ensuring the validity and reliability of research findings, especially in fields where data quality directly influences critical decisions (Van den Broeck, et al.). The table below provides just some of the modern methods for data cleaning provided by the aforementioned sources. Links in the second column reference general articles that break down more of the detail involved in these techniques and how they are implemented.
| Problem Type | Tool/Approach | Solution | Remarks |
|---|---|---|---|
| Illegal values, inconsistencies in a single source | Data Profiling | Detect out-of-range values, identify unusual patterns using cardinality (size), range, frequency | Typically identified through metadata analysis |
| Duplicate records across sources | Approximate Matching Algorithms | Use fuzzy logic or edit-distance to detect near-duplicate records | Works when exact identifiers are unavailable or misspelled |
| Embedded or misfielded values in text | User Defined Functions (UDF) | Extract, split, or reformat structured data from plain text | Often implemented in SQL or custom functions |
| Structure-level conflicts (naming, foundational) | Schema Integration and Translation | Translate structure to uniform format, resolve synonym/homonym issues | Pre-processing required before specific instance cleaning |
| Different representations | Standardization | Unify formats (e.g., date, name casing), resolve abbreviations | Improves matching accuracy across sources |
| High-volume error detection | ETL (extract, transform, load) Tools | Define cleaning processes in workflows with reusable logic | Supports scheduling, dependency tracking, adaptive cleaning |
How AI Reacts to “Bad” Data
When AI encounters bad or biased data, the consequences can be far-reaching and deeply problematic, as described in articles from Harvard Business School and Chapman University. AI systems are inherently dependent on the quality of data they are trained on. When this data contains errors, missing pieces, or social biases, the AI will replicate those flaws and may amplify them. The Harvard article explores a real-world case in which a healthcare organization used AI to automate staff scheduling. The system was trained on historical data that failed to reflect the complexity of human needs and workplace dynamics. Things such as individual preferences, fatigue, and objectivity in workload distribution. As a result, the algorithm produced schedules that were efficient on paper but practically ineffective leading to widespread dissatisfaction and increased turnover among the employees. This situation highlights how AI, lacking a human sense, can magnify inefficiencies and injustices when trained on flawed data (Schrage, Michael) . Meanwhile, the Chapman University article delves into the broader issue of bias in AI, emphasizing how historical inequalities such as racial, gender, and socioeconomic discrimination can be embedded in datasets. When these datasets are used to train models without proper oversight, the AI learns and perpetuates these same inequities. As an example, facial recognition software has been shown to misidentify people of color at far higher rates than white individuals, mostly due to underrepresentation or skewed representation in training data. The article also highlights that even well-intentioned developers have the ability to fail to recognize hidden biases in data, particularly when the datasets are large, complex, or proprietary. In each case, the AI reacts not by correcting for these issues, but by learning patterns blindly and executing them without ethical context or social awareness (Chapman University). This reveals a key limitation of AI that speaks to it not having a human understanding of right or wrong. Instead, its outputs are reflections of the inputs it receives. Without intentional human intervention that include data audits, inclusive data collection, and algorithmic fairness checks AI will continue to reinforce existing societal problems. Therefore, the way AI reacts to bad or biased data is not neutral. It actively contributes to harmful outcomes unless carefully designed to do otherwise. These examples show a need for increased accountability, transparency, and ethical attentiveness in AI development to ensure that the technology serves society equitably.
Cultural Implications
The cultural and social implications of bias and bad data in AI are profound, as these technologies are increasingly intertwined with how societies operate, make decisions, and shape identities. According to UC Santa Barbara’s Dirty Data article, biased datasets can inadvertently encode and reinforce stereotypes, resulting in algorithmic outputs that marginalize certain groups or misrepresent their realities. When predictive algorithms used in criminal justice systems are trained on historically biased policing data, they may allow for over-policing in minority communities. These outcomes not only exacerbate existing inequalities but also legitimize them under the guise of objectivity and automation. The UCSB article frames this issue as an opportunity for intervention, suggesting that bias detection and correction can be integral to ethical AI development, rather than an afterthought (Ortiz). Complementing this, the systematic review by contributors to the African Journal of Biological Sciences emphasizes how AI has become a tool for social transformation, deeply embedded in cultural decision-making and public perception. The authors argue that AI doesn’t merely reflect culture but actively shapes it. Cultural management systems are increasingly reliant on AI to interpret social data and forecast trends. However, when these systems are built on flawed or culturally insensitive data, they risk eroding cultural diversity, distorting collective values, and standardizing perspectives in ways that might privilege dominant ideologies. This is especially troubling in globalized information networks, where AI-driven platforms mediate communication, education, and even artistic expression. The journal highlights the need for human oversight, ethical standards, and inclusive data collection to ensure AI systems support cross-culturalism rather than suppress it (Singh, Abhishek K., et al.). Both sources stress that data is not neutral, it carries the biases of its creators and contexts. Therefore, culturally aware AI development needs to prioritize transparency, societal engagement, and rigorous auditing to prevent digital tools from becoming instruments of social exclusion. In this digital age, where data-driven decisions increasingly shape life, the cultural and social costs of ignoring bias in AI can be as significant as the technological gains it hopes to provide. Ultimately, if left unchecked, AI systems risk becoming active participants in deepening the divisions of our society.
Final Thoughts
As we move deeper into the digital age, the challenge is no longer about the lack of data, but about how to meaningfully engage with an abundance of it. With only a small fraction of global data generated being usable, the importance of data cleaning, relevance, and ethical oversight is understated. Clean and purposeful data forms the foundation for trustworthy AI, while biased or messy datasets can have lasting social and cultural consequences. The technologies we build reflect the assumptions we build them with, sometimes consciously, but often not. If we want to design AI systems that are fair, accountable, and beneficial across societies, we must rethink our relationship with data. This begins with asking not just how much data we have, but whose stories it tells, what it leaves out, and how we choose to interpret it. In doing so, we can move from information overload to insight that informs equitable decisions.
Bibliography
Chapman University. “Bias in AI.” AI Hub, https://www.chapman.edu/ai/bias-in-ai.aspx. Accessed 12 Apr. 2025.
Kirschstein, Timo, and Thomas A. Gern. “Benchmarking AI Code Generators on Programming Problems from LeetCode.” Automated Software Engineering, vol. 31, no. 1, 2024, https://doi.org/10.1007/s10515-024-00453-w.
Muenchen, Robert A., and William L. Bainbridge. “The Impact of Artificial Intelligence on Human Values.” Journal of Evolution and Technology, vol. 13, no. 2, Aug. 2003, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1198040/.
Ortiz, Andrea Estrada. “Dirty Data: An Opportunity for Cleaning up Bias in AI.” The Current, University of California, Santa Barbara, 15 Feb. 2024, https://news.ucsb.edu/2024/021521/dirty-data-opportunity-cleaning-bias-ai.
Rahm, Erhard, and Hong Hai Do. “Data Cleaning: Problems and Current Approaches.” Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 23, no. 4, Dec. 2000, pp. 3–11.
Reinsel, David, John Gantz, and John Rydning. The Digitization of the World: From Edge to Core. IDC White Paper, sponsored by Seagate, May 2020, https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf.
Richter, Felix. “Volume of Data Created, Captured, Copied, and Consumed Worldwide from 2010 to 2025.” Statista, 2024, https://www.statista.com/statistics/871513/worldwide-data-created/.
Schrage, Michael. “Bad Data, Bad Results: When AI Struggles to Create Staff Schedules.” Working Knowledge, Harvard Business School, 6 Nov. 2023, https://www.library.hbs.edu/working-knowledge/bad-data-bad-results-when-ai-struggles-to-create-staff-schedules.
Singh, Abhishek K., et al. “Artificial Intelligence (AI) and Socio-Cultural Management – A Systematic Review of AI’s Effect on Decision-Making.” African Journal of Biological Sciences, vol. 6, no. 12, 2024, pp. 2316–2327, https://doi.org/10.48047/AFJBS.6.12.2024.2316-2327.
Van den Broeck, J., Cunningham, S. A., Eeckels, R., & Herbst, K. (2005). Data cleaning: Detecting, diagnosing, and editing data abnormalities. PLoS Medicine, 2(10), e267. https://doi.org/10.1371/journal.pmed.0020267