How organizations define, discover and protect personal data without losing its value.Data drives growth and innovation, but it has also been monetized in ways that stretch, or even violate, ethical boundaries: used without regard for privacy or informed consent and, at its extremes, to shape opinions and influence behavior. That is why responsible data governance and stronger regulatory oversight matter, so data can serve society without compromising trust or individual rights. This article takes a closer look at another critical dimension of data governance — Data Privacy & Regulatory Compliance.Image by author — Dimension — Data Privacy & Regulatory ComplianceData privacy and regulatory compliance standards are designed to helporganizations meet global data protection, privacy, and security obligations. Modern data platforms align their controls with widelyrecognized international frameworks and regulations, including GDPR and CCPA. These standards emphasize strong governance of customer data, ensuring that personal and sensitive information is processed lawfully and transparently, and for clearly defined purposes.Customers retain ownership and control of their data, while the data platform acts as a data processors under applicable privacy laws.From awareness to accountabilityEarlier regulations, such as HIPAA, drew attention to the need to protect personal data, but enforcement was often limited and inconsistent. That changed in 2018, when the European Union enacted the GDPR. It redefined standards for the ethical collection, storage and use of personal data, and backed them with substantial fines and penalties. Individuals now have strong rights over their data: to be informed, to access and correct it, to request deletion in certain cases, to object to processing for marketing or profiling, and to opt out of its sale or sharing.What counts as PII?Personally Identifiable Information (PII) is any information that can be used, on its own or combined with other data, to identify a specific individual. Direct identifiers include government-issued IDs such as passport or Social Security numbers; indirect identifiers include email addresses, phone numbers, IP addresses and location data that can be linked to a person. Organizations must ensure lawful collection, purpose limitation, secure storage, restricted access and proper disposal of PII throughout its lifecycle.PII Lifecycle: Define, Discover, HandlePII lifecycle is the process of identifying and locating PII across an organization’s data landscape. Its goal is to answer three critical questions:Where does the PII live?What type of PII exists (direct or indirect)?How it is being used?The PII lifecycle includes scanning structured data such as databases, data warehouses, data lakes, and lakehouses, as well as unstructured data like documents, logs, emails, and files stored in cloud platforms. Effective PIIhandling creates visibility into sensitive data that may otherwise be hidden or inconsistently labeled. The following stages represent the recommended lifecycle for handling Personally Identifiable Information (PII).Define : In this stage, the Data Protection Officer, Privacy Office, Legal and Compliance teams, and Information Security offices agree on what counts as direct and indirect identifiers. In a federated model, these central standards flow to business domains, where data owners and stewards identify, classify and label PII in their own datasets.Discover : Finding where PII lives, what type it is, and how it is being used. Automated techniques such as pattern matching, machine-learning classifiers and metadata analysis scan structured and unstructured data, and data owners and stewards validate the results. PII discovery typically combines automated techniques — such as pattern matching, keyword searches, machine-learning classifiers, and metadata analysis — with human validation by data owners and stewards. Following is a list of some well-known tools and services that can automate the PII discovery process:Microsoft Purview: Built-in PII discovery and classification across Azure environments.AWS Macie: PII and sensitive data discovery for Amazon S3Google Cloud DLP: Pattern- and ML-based PII detection across GCP servicesDatabricks Unity Catalog: Column-level lineage, tagging, and integration with PII detection toolsBigID: ML-driven discovery of PII, sensitive data, and data relationships across on-prem and cloudCollibra Protect: Integrates governance metadata with sensitive data discoveryInformatica Data Privacy Management: Enterprise-grade PII discovery,classification, and risk scoringHandle : Once PII is discovered, several techniques including masking, anonymization, pseudonymization & tokenization are used to reduce privacy risk while still enabling analysis, testing and sharing. The table below summarizes some well-known techniques for handling PII.Image by author — PII handling techniquesFor a detailed explanation of how each PII handling method is implemented, I recommend reading my previous article below:https://medium.com/aws-in-plain-english/regulatory-compliance-gdpr-and-ccpa-using-delta-lake-fa0582b2fc2dIt is important to note that while the methods discussed above effectively obscure PII through self-coded data transformation techniques, they can sometimes be complex and challenging to implement. Fortunately, there is a simpler yet highly effective alternative:Obscuring PII using masking functionsIn addition to the techniques discussed above, PII obscuring using functions is a widely adopted approach that relies on custom or built-in transformation functions to reduce the exposure of sensitive data while maintaining usability for analytics, testing, or operational purposes. These functions can be applied at ingestion time, during data transformationpipelines, or at query time, enabling organizations to protect PII without physically removing it from the underlying dataset.Most modern data platform provide data masking functions that replace all or part of a sensitive value with fixed or dynamic patterns — for example, displaying only the last four digits of a Social Security Number or credit card number. Function-based PII obscuring is particularly effective when combined with data classification, role-based access controls, and policy enforcement, ensuring that only authorized users can view clear-text values while others see masked, hashed, or tokenized data.Image by author — Data masking functionsThis approach helps organizations meet privacy and compliance requirements, reduce risk, and preserve analytical value, making it a core component of modern data governance and privacy-by-design strategies.Each PII handling technique offers a different balance between data utility and privacy protection, and their suitability depends on factors such as regulatory requirements, intended use, and risk tolerance. Because anonymization irreversibly removes the ability to identify individuals, while pseudonymization retains a controlled link to original identities, each method has distinct advantages and limitations.Rather than prescribing a single approach, this article presents all techniques so readers can evaluate and select the one that best aligns with their PII-handling objectives, compliance obligations, and operational needs.In addition to PII techniques discussed above, there is another approach to managing sensitive data that offers greater flexibility — enabling organizations to control how much data is exposed, rather than choosing between fully hiding it and exposing it entirely — Data Classification Policies. We will talk about it in greater detail in the upcoming article.I hope this article offered a clear primer on the first the key dimension of data governance. As promised, I’ll explore more dimensions in greater depth in the upcoming articles, so stay tuned. Topics like these are also covered in the Governance, Risk and Compliance (GRC) Specialist course offered through Datafence Cloud Academy.

Ensuring the right people have access to the right data at the right time with the right controls.In the last article, we introduced the key dimensions of data governance and discussed their importance in building a robust governance framework. I also promised that we would explore each dimension in greater depth in future articles. This article delivers on that promise and takes a closer look at one of those critical dimensions — Data Access Policies.Image by author — Dimension — Data Access PolicyAt a high level, data access policies define who can access which data, at what level of detail, and under what conditions. These policies ensure that data is used securely and responsibly, in compliance with organizational standards and regulatory requirements.The four questions every data access policy needs to answerWho: Defines the authorized users, groups, roles, or service accounts that are permitted to access data. This is typically aligned with business domains or functions and supports access models such as role-based access control (RBAC) and attribute-based access control (ABAC).What data: Specifies the data assets that can be accessed, including catalogs, schemas, tables, views, files, or models. Access is granted only to datasets that are relevant to a user’s role or business responsibility.Level of detail: Determines the granularity of access, such as read-only versus read-write permissions, row-level filtering, and column-level visibility or masking for sensitive information such as PII.Under which conditions: Establishes contextual access constraints, such as the environment (development, test, or production), and compliance, audit, legal, or policy-driven requirements.Image by author — Data Access PolicyTo whom are data access policies granted ?Modern data platforms manage access through identities, including users, groups, and service principals. To maintain a scalable and governable access model, these identities should be structured around business domains and organizational responsibilities. Furthermore, access permissions should be granted at the group level rather than directly to individual users. This simplifies administration, promotes consistency, and makes it easier to enforce least-privilege access principles across the organization.Image by author — Granting data access policies to users, groups, and service principalsHow are data access policies implemented?The primary objective of data access policies is to enforce the principle of least privilege, ensuring that users are granted only the minimum level of access required to perform their responsibilities. This approach significantly reduces the risk of unauthorized access and data exposure.The principle of least privilege states that users, roles, and systems should be granted only the minimum access necessary to perform their authorized tasks. This reduces security risk by limiting unnecessary exposure to data, systems, and sensitive operations.Data Access Control ModelsData access control models provide the framework for implementing data access policies — assigning, managing, and governing permissions to ensure that users can access only the data and resources they are authorized to use. Following are some well-known data acccess cotrol models:Access control lists (ACL)ACL is a resource-level security model that specifies which users, groups, or service principals are authorized to access a particular resource (tables, views etc.) and what operations they can perform. Because permissions are assigned directly to each resource, ACLs provide a simple and intuitive access control model. However, as the number of resources and identities increases, managing ACLs can become complex, time-consuming, and prone to inconsistencies.Role-based access control (RBAC)In the RBAC model, permissions are assigned to roles that represent specific job functions, and users gain access through their assigned roles. This approach improves scalability, simplifies administration, and supports auditing and governance. However, organizations must carefully manage role design to avoid role explosion, where an excessive number of roles becomes difficult to maintain.Fine‑grained access control (FGAC)Fine‑grained access control (FGAC) is a security authorization model that governs access to data assets across multiple hierarchical scopes, including the catalog, schema, and individual securable objects such as tables, views, functions, and columns. FGAC enables precise, object‑level permission management by explicitly defining and enforcing privileges based on the operation type (e.g., SELECT, INSERT, UPDATE, DELETE, EXECUTE, ADMIN).Under FGAC, access can be progressively constrained. For example, a principal (user, group, or service identity) may be granted read‑only access at the catalog level, restricted access to specific schemas within that catalog, and further limited to a subset of tables or columns within a schema. This hierarchical and explicit grant model ensures tight control over data exposure and usage.Attribute-based access control (ABAC)ABAC is a governance model that enforces policies such as row-level filtering to dynamically hide unauthorized rows and column-level masking to obfuscate sensitive fields. These controls are applied at query execution time using governed tags. These tags are assessed against defined access policies, allowing the system to dynamically filter, mask, or exclude sensitive information — such as PII — based on the user’s attributes, role, and context.Image by author — ABAC enforcing row and column level controlsIt is important to understand that while ABAC defines policies using governed tags, FGAC provides an enforcement engine that evaluates these policies at query time.To sum up, data access policies should be centered on a group‑based, least‑privilege security model, with groups preferably aligned to business domains and clearly defined access levels to ensure scalable, auditable, and well‑governed access.I hope this article offered a clear primer on the first the key dimension of data governance. As promised, I’ll explore each dimension in greater depth in the upcoming articles, so stay tuned. Topics like these are also covered in the Governance, Risk and Compliance (GRC) Specialist course offered through Datafence Cloud Academy.

Evolving data assets into governed, trustworthy data products that create consistent and measurable value.Lately, I’ve noticed a clear shift across the industry — organizations are moving away from simple data assets that merely deliver value, toward fully governed data products that deliver value with confidence. This transition isn’t just about packaging data differently; it’s about elevating trust, reliability, accountability, and usability so data becomes a dependable product rather than a loose collection of tables.But the path to strong data governance is long and demanding — governed data products requires discipline, maturity, and a clear framework. A well-designed governance strategy not only safeguards organizational data assets but also enables organizations to fully realize the value of analytics by fostering trust among business users and providing assurance to security, risk, and compliance stakeholders.It’s not just a technical journey; it’s an organizational transformation that touches people, processes, and culture.Implementing a robust data governance strategyAt its core, data governance brings together a set of complementary practices that collectively establish trust, protect sensitive information, and ensure consistent, responsible use of data across the organization. When implemented effectively, these practices form a cohesive framework that balances control with enablement, allowing data products to be shared confidently and consumed at scale.So how does one begin tackling the challenge?Data governance strategy typically encompasses several key areas, each contributing to the organization’s ability to safeguard data while maximizing its analytical value. Together, the governance dimensions as defined in the image below define how data is owned, secured, classified, certified, monitored, and evolved over time — ensuring that users can trust not only the data itself, but also the processes and controls that support it.Image by author — Data governance standards and policiesToday, let’s take a high‑level look at how each governance dimension plays a role in keeping data secure, trustworthy, and compliant with regulatory requirements. And don’t worry — we’ll dive deep into every one of these dimensions in the articles that follow.Data access policiesAt a high level, access policies define who can access which data, at what level of detail, and under what conditions. These policies ensure that data is used securely and responsibly, in compliance with organizational standards and regulatory requirements.Data privacy and regulatory compliance standardsData privacy and regulatory compliance standards ensure that organizations meet global data protection requirements — such as GDPR and CCPA — while upholding strict obligations around privacy, security, and responsible data handling.Data classification policiesData classification policies impose a systematic process of organizing data into defined categories based on factors such as its sensitivity, business value, regulatory impact, and risk level. By classifying data, organizations can determine who should have access to it, how it must be protected, and which policies and controls apply.Data lineage tracking policiesData lineage is the practice of tracking how data is used as it flows through systems — from where it’s created to where it’s consumed. It involves systematic documentation of data origins, flow, dependencies, transformations, and downstream consumption across systems throughout the data lifecycle.Data lifecycle management policiesData lifecycle management policy is a formal framework that establishes how long an organization retains information, the purpose for retaining it, where it is stored, and the conditions and methods for its secure disposal.Data quality standardsAmong the four Vs of big data, data veracity emphasizes the key dimensions of accuracy, trustworthiness, and reliability. Data quality standards measure how well data aligns with key business objectives based on these dimensions.Data observability standardsData observability standards define how organizations monitor and ensure the health, reliability, and trustworthiness of data as it moves through modern data pipelines and ultimately powers data products.Data stewardshipData stewardship is the human accountability layer of data governance — the set of roles and responsibilities that ensure data is accurate, trusted, well-understood, and used appropriately across an organization.In short, data governance policies define the rules of engagement, and data stewards turn those rules into reality.I hope this article offered a clear primer on the key dimensions of data governance. As promised, I’ll explore each dimension in greater depth in the upcoming articles, so stay tuned. Topics like these are also covered in the Governance, Risk and Compliance (GRC) Specialist course offered through Datafence Cloud Academy.

A comprehensive guide covering Lakehouse architecture, Delta Lake, Unity Catalog governance, data quality frameworks, secure data product design, and practical GenAI integration for enterprise analytics.In an era where data has become the backbone of every modern enterprise, organizations are under immense pressure to build platforms that are not only scalable and high‑performing but also governed, secure, and trustworthy. The rapid evolution of cloud technologies, the rise of Lakehouse architectures, and the explosive growth of GenAI have created both unprecedented opportunities and significant risks. This is the landscape in which Building Trusted Data Platforms with Azure Databricks and GenAI has become an essential guide for data leaders, architects, and engineering teams worldwide.A Book Built for the Modern Data RealityMost enterprises today struggle with fragmented data systems, inconsistent governance, unreliable pipelines, and unclear ownership models. As organizations adopt Databricks, Delta Lake, Unity Catalog, and AI/ML workloads, they often lack a unified blueprint for building platforms that are both innovative and compliant.My book fills that gap.Image by authorIt provides a practical, hands‑on framework for designing Lakehouse architectures that balance agility with governance, performance with cost efficiency, and innovation with regulatory responsibility. It is one of the few resources that connects data engineering, governance, security, and GenAI into a single, coherent operating model.Governance Is No Longer OptionalWith increasing regulatory pressure — GDPR, CCPA, PCI, HIPAA, ISO, SOC — organizations must treat governance as a first‑class engineering requirement. Your book shows how Unity Catalog, Purview, lineage, quality frameworks, and domain‑driven ownership can be embedded directly into Databricks Lakehouse design.Growing Demand For Trusted DataAI and GenAI systems are only as reliable as the data products that feed them. As enterprises accelerate adoption of LLMs, agentic workflows, and AI‑driven analytics, the need for high‑quality, well‑governed, domain‑owned data products becomes critical. GenAI models depend on consistent schemas, accurate metadata, strong lineage, and trustworthy semantics to produce reliable outputs. Poorly governed data products lead to hallucinations, biased predictions, compliance violations, and operational risk. My book emphasizes the foundational role of trusted data products — built with Delta Lake, Unity Catalog, quality checks, lineage, and domain stewardship — as the backbone of safe, scalable, and responsible AI. By ensuring that every data product is discoverable, certified high‑quality, and compliant, organizations can confidently power GenAI workloads that are accurate, explainable, and enterprise‑ready.What does a Futuristic Data Platform Look Like?Growing demands of today’s data analytics requires a next-generation, future-ready data platform that transcends traditional analytics to become the intelligent backbone of the organization should embody the following characteristics:Image by author — Hallmarks of future ready data platformWhy are the blocks structured in the shape of a pyramid ?Each block in the data platform strengthens the next until data transforms into actions. Together, these layers create a cohesive technology stack in which each layer serves as the foundation for the one above it. Missing a single layer weakens the structure, and the pyramid loses its balance. To understand how the pyramid achieves and maintains its balance.The evolution of future data platforms is a re-imagining of data-driven decision-making. It’s a re-conceptualization of the journey of data — production, transformation, distribution, and consumption. Modern data platforms are quickly becoming corporate growth enablers, increasinglydefining the core difference between organizational success and failure.A well-designed future data platform represents a convergence of data-driven intelligence, modern data architecture, cutting-edge cloud-enabled services and frameworks, automation, and operational excellence.The real question is: what are the defining hallmarks of a truly futuristic and cutting-edge data platform?Uses an easy-to-use, scalable base infrastructure (preferably using the cloud) to sustain critical loads with optimal performance.A data architecture that supports both batch and streaming workloads, adapts to varying data formats, and supports the ever-increasing demands of consumers.Deliver a certain level of trust to its users by delivering federated governance, high data quality, and certified data products.Facilitate a secure and seamless mechanism for distributing data across diverse internal domains and external entities.Creates a centralized repository that addresses core MDM challenges for key data entities, including duplication, inconsistency, and accuracy issues.Leverage automation to streamline infrastructure and code deployments.Ensure health, visibility, and reliability of data using trustworthy data observability practices.Follow ethical AI standards by adhering to industry best practices.Together, these elements form the blueprint for future-ready data platforms — systems that are not only well-governed, resilient, and intelligent, but also adaptable to continuous change. They enable organizations to operate with confidence in an environment defined by rapid data growth, evolving regulatory expectations, and accelerating advances in AI.Overall, a next-generation, future-ready data platform transcendstraditional analytics to become the intelligent backbone of any organization.I wasn’t able to cover every topic above in depth, but if you found this material valuable, you’ll benefit even more from my book, where each concept is explored thoroughly with hands‑on guidance and real‑world architectural patterns.I hope this article has given you clarity and direction on the emerging trends shaping the future of data platforms. These concepts — along with a deeper exploration of modern Lakehouse patterns, Databricks best practices, and GenAI‑driven architectures — are covered extensively in my Databricks Certified Data Engineer and GenAI Engineer course offered through Datafence Cloud Academy, which I teach live online on weekends for practitioners looking to advance their expertise.Thanks for reading.

In this article, we will learn how Amazon DataZone promotes the idea of “self-serve data analytics”. by simplifying the data collaboration…Continue reading on AWS in Plain English »

In this article, we will learn how to build the “Golden Record” in MDM using an unsupervised linking model Splink embedded within…Continue reading on AWS in Plain English »

In this article, we will focus on some key methods to effectively optimize your AWS cloud resource costs.Continue reading on AWS in Plain English »

In this article, we will try to understand how the output from Delta Lake change feed can be used to feed downstream applicationsContinue reading on TDS Archive »

In this article, we will try to understand how we can handle regulations such as GDPR and CCPA using Delta LakeIn the world we live in, the power of data cannot be challenged. This power has helped several companies accelerate their revenue by monetizing and marketing personal data in ways that cross the boundaries of ethical use. In some cases, the power of data has been misused to the extent of altering human behavior in such a way that it favors one candidate over others in ongoing elections. The whole idea of democracy is being challenged.In the past known regulations such as Health Insurance Portability and Accountability Act (HIPAA) started to sharply focus on the importance of protecting Personal Identifiable Information (PII). The problem is that even though the regulation was enacted, it was rarely implemented or enforced. In 2018, the General Data Protection Regulation (GDPR) was passed by the European Union. In many ways, GDPR has not only rewritten the rule book on ethical collection, storage, and usage of private data but has taken a strong stance on violations by imposing harsh fines and penalties. Recently, California Consumer Privacy Act (CCPA) came out with a similar regulation to prevent the over-collection, sale, and usage of personal data. The question is what kind of information are these regulations trying to protect?In broad terms, PII is any piece of information that can be used to link (directly or indirectly) back to a consumer. In other words, it can be used to target consumers for usages such as marketing and influencing against their will.Let us try to understand the fundamental rights granted to consumers under GDPR and CCPA and attempt to cover each aspect using the Delta Lake framework.In the article below we will try to understand the following techniques related to data anonymization and pseudonymization. Each technique has its own advantages and disadvantages so I will leave it up to the reader to decide which one to use.Image by authorRight of AccessThe consumer has the right to request a copy of their personal data. They also have the right to know how their data is being stored, retained, processed, and shared. In addition to this, they also have the right to know if their personal data was used in automated decision-making processes. This information needs to be sent to the consumer in an electronic form. Most organizations these days allow consumers to download this information from their websites.One of the key requirements of GDPR and CCPA is to employ the highest-grade security mechanisms to protect PII data. In the sections below we will discuss the available options to comply with this requirement mostly in the context of storing and retaining consumer information. In the following sections, we will learn about the commonly used techniques for complying with GDPR and CCPA requirements.Kick-Start the Governance ProcessThe recommended starting point for protecting PII data starts by clearly establishing the list of PII columns within your dataset. Creating an accurate list of PII columns requires a combination of manual and automated identification techniques. In addition to working with internal teams in charge of governance within an organization, you may use services like Azure Purview and AWS Glue DataBrew that can automatically identify PII columns for you. Once the list has been created you may store the list as a control table for further use.In the example below, we will use the storage terminology of Lakehouse architecture. If you are not familiar with the Lakehouse architecture you may read this article.Our example dataset includes the following list of PII columns:Image by authorNow let us handle a few GDPR and CCPA scenarios as under:Scenario 1 — Anonymization of PII Data using Data MaskingData Masking is the most common anonymization technique that can be used to obscure PII data. Data masking is usually implemented using a one-way cryptographic function known as hashing. You can create a hash from the cleartext information but not the other way around, in other words once de-identified it cannot be re-identified to its original form. But cryptographic hashes are not completely immune to hacks, they can be defeated using brute force or dictionary attacks. For this reason, it is common to use salted hashes. This is how it works.Image by authorWe can start the process by generating an employee-salt-key control table for storing salt keys per employee. The following Spark UDF implements the actual masking of the bronze layer dataframe.Image by authorThe code below will transparently masks all columns in the employee dataset that match the PII control list (PIIControlList). Employee data is read from the source in a Spark dataframe (employeeBronzeDF) and joined to the employee-salt-key table so that the appropriate salt key is used for the hashing process. This randomizes data during the masking process to make it withstand brute force or dictionary attacks.Image by authorFinally, the masked dataframe is written as employee-silver-masked Delta table in the silver layer.Scenario 2 — Pseudonymization of PII Data using Data EncryptionData encryption is a pseudonymization technique that can be used to obscure PII data. Data encryption is a two-way cryptographic function whereby it is not only possible to de-identify PII data but also reverse or re-identify it back to its original form. A core requirement for implementing encryption is the creation and safe management of a symmetric/asymmetric key. An easy way to do this is by using a key management service from the cloud vendors. The cloud vendor will then make sure they protect the confidentiality, availability, and prevention of unauthorized use. For this article, we have used the AWS Key Management Service (KMS).To create a new encryption key, log on to the AWS Portal. Browse to AWS KMS > Customer Managed keys > Create Key. Now create a new key that looks like this:Image by authorIf you create the key with the same alias as above, then simply need to reference your AWS access keys in the notebook. Get access to the AWS access key using the link below.https://console.aws.amazon.com/iam/home?region=us-east-1#/security_credentialsThe access key (access key ID and secret access key) will be used as the credentials for the Databricks notebook to access AWS services.Image by authorAfter the encryption key has been set up, we are all set to run the encryption process. The following functions will be used to perform the encryption. Note that this function will encrypt only the columns that are referenced in the PII control list (PIIControlList).Image by authorFinally, the code below will encrypt columns in the employee dataset that match the PII control list (PIIControlList).Image by authorSince data encryption is a two-way cryptographic function, we can easily re-identify data back to its original form. The following functions can be used to reverse the effect:Image by authorUsing the functions above we are able to perform the reversal of pseudonymization.Image by authorScenario 3 — Pseudonymization of PII Data by Splitting TablesAnother commonly used pseudonymization technique can be implemented by splitting the PII and non-PII data into two separate tables that are connected using a pseudonym key. This is how it can be implemented using Delta Lake.Image by authorThe code above splits the employee data into two separate tables (employee-silver-pii & employee-silver-non-pii) connected using an email pseudonym key. Using the security in layers approach, the organization may choose to go a step further by encrypting the table that contains PII data (employee-silver-pii). Using this scheme, it is important to properly secure access to the table that contains PII data.Image by authorScenario 4 — Pseudonymization of PII Data using TokenizationThe final pseudonymization technique that we will discuss is using tokens. The implementation of tokenization involves the replacement of columns containing PII data values with tokens (typically a placeholder). The actual value of the placeholder is stored in another table, therefore making the real values (PII data) obscured. Let’s find out how this is done. The code below reads employee data and generates tokens for data and creates a token for each column relating to the employee row.Image by authorOnce the tokens have been generated, they are replaced in the actual table that contains PII information.Image by authorUsing this method, the PII tokens may be replaced anytime with actual values by performing a reverse operation.Right of Rectification and ErasureThe “right of access” further grants the consumer to exercise several other rights for controlling how their data gets used.Right of rectification — The consumer has the right to request rectification of personal data if it is inaccurate.Right of erasure — Consumer has the right to request their personal data to be erased.This is how Delta Lake can help you comply with these types of requests.Scenario 5 — Comply with Data Rectification RequestsHere is a case where an employee has requested a change of their PII information:Image by authorAll we need to do is merge the change data into the existing delta lake table as follows:Image by authorScenario 6 — Comply with Data Deletion RequestsSimilarly, the following employee has requested their record be deleted. This is how you can comply with the delete request.Image by authorNotice how delta lake can perform atomic deletes on existing tables. Once the employee record has been deleted, we have intentionally performed a vacuum operation. The vacuum operation in delta lake removes data files that are not in the latest state of the transaction log. Using a retention time of 0 hours ensures that the employee record is immediately expunged from history as well.Image by authorThe pseudonymization and anonymization scenarios described above should hopefully provide readers with enough information to choose the best technique as per requirements.For any organization, the journey to effectively deal with the GDPR and CCPA requirements starts by storing data in a format like Delta that supports obfuscation of PII data and allows for atomic updates and deletes.Additionally, the organization needs to maintain a strict data retention, data lineage, and classification policy. However, the organization still needs to be cautious about PII data in other data layers such as bronze, source systems, or backups. Therefore, it is recommended to create an enterprise plan that clearly addresses the protection and privacy of both “data-at-rest” as well as “data-in-motion”.More content at PlainEnglish.io.Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.Build awareness and adoption for your tech startup with Circuit.

Handling the challenge of slowly changing dimensions using the Delta FrameworkContinue reading on TDS Archive »