Python Data Engineering Interview: 16 Practical Questions

Read incrementally and avoid collecting every parsed record in a list. For a line-oriented input, iterate over the file; for structured formats, use a parser that supports streaming or batches. A generator can yield one transformed item at a time, but downstream code can still materialise the whole result. Identify which operation, such as sorting or a global join, needs additional storage.

Python functional programming guide

CSV can contain quoted delimiters, embedded newlines and escaped quotes. Python’s csv module handles configured dialects; a simple split does not. Open CSV files with newline="" as documented and choose an encoding deliberately. Parsing a row does not validate its business meaning, so separately check required columns, identifiers and numeric fields.

Python csv documentation

Define whether the field is required and which representations are accepted. Parse explicitly, catch the expected conversion error and record the row identifier and reason for rejection. Do not silently turn every failure into zero: that can alter totals. Explain whether the job rejects the file, quarantines the row or continues under an agreed threshold.

Use a list when order and repeated items matter, a dictionary for key-based lookup, and a set for membership or uniqueness when the values are hashable. Choose the key carefully: identical names are not necessarily duplicate customers. Discuss memory use and the output-order requirement before replacing a sequence with a set.

Start with the business grain: one event, order or customer version. Define a stable key and a rule for conflicting versions. A repeated order ID may be a retry, an update or bad data. Keep enough lineage to explain which record was retained. Deduplicating complete rows will not resolve two versions with different timestamps or amounts.

Repeating the same logical operation should not create an additional business effect. For an import, use a stable source key and a destination write strategy that recognises an already applied item. Record completion only after the intended write commits. Retrying a request is not automatically safe merely because the surrounding Python function has the same arguments.

Compare the incoming schema with an explicit contract: required fields, types and allowed changes. An extra optional column may be acceptable; a renamed identifier may require stopping the load. Log the detected difference and retain a recoverable input reference. Do not let positional column mapping silently shift values into the wrong destination fields.

Distinguish event time from arrival time and define how far back the job revisits data. Use stable keys so overlapping windows can be processed without duplicating effects. Explain what happens to an event outside the normal window: a correction path, a backfill or an exception. A maximum timestamp alone can miss delayed or updated source records.

Retry only when the error and operation make it appropriate. Use bounded attempts and respect the service’s retry guidance. A timeout does not prove the server did nothing. For a write, use the API’s supported idempotency mechanism or reconcile the result before retrying. Authentication and validation failures usually require correction rather than repeated identical requests.

IETF HTTP semantics

Follow the documented cursor or next-page mechanism until it indicates completion. Save progress at a safe commit boundary and define handling for expired cursors or changed data. Test an empty page, a repeated cursor and a failure between pages. Page numbers alone do not guarantee a consistent snapshot when the source changes during extraction.

Pass values separately using the database driver’s parameter mechanism. With sqlite3, use placeholders and a parameter sequence instead of concatenating user input into the SQL text. Placeholder syntax differs by driver. Table or column identifiers need a separate controlled selection strategy; they are not ordinary value parameters.

Python sqlite3 parameter binding

Describe the transaction boundary first. If the entire batch is atomic, roll it back and retry safely after addressing the cause. If records commit independently, retain per-record completion information and resume without duplicating successful writes. Test the interruption between destination commit and checkpoint update, because that gap can produce a repeat on recovery.

Choose based on the workload, libraries and interpreter build. Threads can help overlap waiting on I/O; processes can run work independently but add communication and startup costs. Benchmark representative data and bound concurrency so the source or database is not overwhelmed. Avoid universal claims about Python threading without stating the environment.

Python concurrent.futures documentation

Use small inputs with known outputs: an empty batch, missing fields, duplicate keys, boundary dates and invalid types. Check both accepted data and rejected-row reporting. Include invariants such as total preservation when the transformation should only regroup records. A row-count match alone cannot detect wrong values or a many-to-many join that later gets filtered.

A mock can isolate code and exercise a timeout or error response predictably. It does not prove the real service matches the assumption built into the mock. Keep focused unit checks and add an integration check against the actual interface where practical. Use realistic response shapes and verify the behaviour that matters, not every internal call.

Python unittest.mock documentation

Measure time and resource use by stage before changing code. Compare source fetch, parsing, transformation and destination writes on a representative workload. Check whether batching, a query plan or unnecessary repeated work dominates. Record the baseline, changed condition and result. A faster small fixture does not establish that the production workload improved.