Best Practices for Secure Data Engineering
Who controls which data
The story: Back to the five ways to get a meal (food court, office catering, meal kit, adjusted meal kit, grow your own). Whatever you order is always your choice. The special ingredients you add to change a recipe are yours only when you adjust a meal kit. The farm the base ingredients came from belongs to the supplier, unless you grow your own.
In AI/AWS terms: By Scoping Matrix scope:
| Meal part | Data | What it is | Controlled by |
|---|---|---|---|
| What you order | User data | Inputs from end users | The customer, in every scope |
| Ingredients you add | Fine-tuning data | Data used to adapt a pre-trained model | The provider in Scopes 1–2, the customer in Scope 4 |
| The farm | Training data | The large dataset used to pre-train the model | The provider in Scopes 1–4, the customer in Scope 5 |
For the exam: User data is always the customer's. Fine-tuning data is the customer's in Scope 4. Training data is the customer's only in Scope 5.
Data engineering lifecycle
The story: A kitchen's supply cycle: bring ingredients in, prep them, taste, cook, then adjust the next order based on how it went. Deliveries arrive by several trucks: one that streams in fresh produce constantly, one that moves stock over from the old kitchen, and one that sorts and prepares everything as it arrives.
In AI/AWS terms: The data engineering lifecycle is an iterative cycle of collecting, preparing, and analyzing data to train, evaluate, and improve models. Collection tools on AWS include Amazon Kinesis (streaming), AWS Database Migration Service (moving databases), and AWS Glue (preparing and transforming data).
For the exam: Kinesis, Database Migration Service, and Glue are data collection tools.
Secure data engineering practices
The story: Keeping the supply chain safe: check the quality of every delivery, hide customers' names on orders, lock the stockroom, and make sure nothing is swapped or lost. That last part means checking labels match, keeping backups of the inventory, recording each transfer as a single complete step (never half-done), keeping a log of every movement, and practicing these checks regularly.
In AI/AWS terms:
- Assess data quality
- Use privacy-enhancing technologies
- Control data access
- Ensure data integrity: validation checks (schema, referential, business rules), backup and recovery, transaction atomicity, lineage and audit trails, and regular testing of these controls
Washing, removing duplicate items, and dividing ingredients into batches are kitchen prep, not security. In the same way, cleaning, deduplicating, and splitting data are data preparation steps, not security practices.
The AWS Privacy Reference Architecture (AWS PRA) is the blueprint for designing privacy controls on AWS.
For the exam: Secure data engineering = quality, privacy-enhancing technologies, access control, integrity. Cleaning and splitting are data preparation, not security.