Data Engineer – Unstructured Data & AI Pipelines (Part‑time)

About the role

Neurons Lab is looking for a part‑time Data Engineer / Data Scientist to build the data plumbing for a European private investment group. The project focuses on ingesting, normalising and enriching unstructured data such as calls, emails, Slack messages, documents and historic archives, then exposing it through AI‑enabled retrieval pipelines.

Key responsibilities

• Implement capture pipelines for live calls, emails, Slack and other messengers, with speaker attribution and opt‑out controls.
• Back‑fill years of historic communications and documents, handling parsing, deduplication and correct dating.
• Develop document parsing for PDFs, scanned packs, spreadsheets and slide decks.
• Design and maintain entity‑resolution and record‑linkage across multiple identifiers.
• Build chunking, embedding and vector‑graph storage pipelines using pgvector, OpenSearch or Pinecone‑style services.
• Orchestrate incremental syncs via connector layers, avoiding full re‑crawls and handling edits/deletions.
• Apply PII detection, redaction and retention policies, providing audit evidence to security teams.
• Manage pipelines with Airflow or Step Functions, ensuring idempotency, monitoring and alerting.

Required profile

• 4+ years of data‑engineering experience, especially with unstructured or semi‑structured data.
• Proven ability to integrate multiple third‑party APIs (Google Workspace, M365, Slack, CRM) and perform historical backfills.
• Experience building pipelines that feed LLM or retrieval‑augmented generation systems.
• Comfortable handling sensitive personal data under GDPR and financial‑services security expectations.

Required skills

• Python
• SQL
• Airflow or equivalent orchestration tool
• AWS and/or GCP services
• Vector stores (pgvector, OpenSearch, Pinecone‑class)
• API integration for Google Workspace, Microsoft 365, Slack, CRM systems
• Entity resolution / record linkage (deterministic and fuzzy)
• PII detection, redaction, encryption and retention handling
• Data lineage, provenance and audit pattern implementation