Author(s): Rashmi

Originally published on Towards AI.

Why data science needs a different playbook

Data science work has a structural problem that ordinary software engineering doesn’t: the artifact that matters most — the dataset — usually can’t live in context, can’t be diffed like code, and changes meaning depending on who’s interpreting it. A data scientist’s day swings between exploratory analysis that’s deliberately messy, model training that’s slow and stochastic, and a handful of moments where the output has to be rigorous enough to inform a real business decision or ship to production. Most AI coding tools are tuned for the second half of that spectrum — clean, deterministic, testable code — and treat notebooks, warehouses, and half-finished EDA as an afterthought.

After the lead, the article explains how Claude Code maps core software-engineering primitives—tool access, subagents, hooks, persistent memory, and MCP connectivity—onto common data-science pain points. It then walks through practical patterns: notebook-native exploratory analysis via NotebookEdit, direct warehouse/database querying through MCP, a dedicated data-analysis subagent to preserve context, data-quality gates enforced by hooks before training or writes, and dataset memory for reproducibility via CLAUDE.md plus persistent agent memory. It covers least-privilege sandboxing for sensitive data, skills to standardize statistical and stylistic conventions, and headless pipelines for retraining and batch scoring. Finally, it proposes an automated weekly insight pipeline that combines these pieces, highlights anti-patterns that break trust (missing validation, overly broad access, non-reproducible outputs, unchecked evaluations), and concludes with future trends like closed-loop experimentation and governance-driven enterprise adoption—arguing that constraints and verification are what make agentic outputs reliable enough to act on.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI