Turning Incompatible Data Streams into a Reliable Product Capability
I designed and built a standalone Node.js ingestion service that collected daily files from secure SFTP sources, translated incompatible schemas into the platform’s internal model, and eliminated manual conversion for the supported dispute-data pipelines.
In 2022, the company had opportunities to support clients whose dispute data came from reporting systems the platform could not ingest directly. The files were available through secure daily transfer, but their structures did not match the product’s existing ingestion logic or internal data models. The differences went beyond field names: records contained missing values, nested structures, inconsistent formats, and data points with no direct equivalent in the platform. Before this project, an unsupported format either created a burdensome, error-prone manual conversion process or blocked onboarding altogether. The business needed a dependable way to translate unfamiliar external data into records the existing dispute-management system could trust and use.
The system at a glance
From incompatible source files to usable dispute records
These diagrams use fictional labels and structures to show the design without exposing client data, provider schemas, credentials, or internal endpoints.
Architecture
A controlled path from secure collection to product workflows
Secure sources
Standalone Node.js service
Platform API
Existing workflows
Schema judgment
Not every external field deserves a permanent internal key
Compare meaning across both models—not just field names.
Operational resilience
Failures stayed visible and stopped before production ingestion
Questionable data never quietly entered the platform.
Business impact
A technical integration removed an operational blocker
Before
Unsupported reporting systems
- Manual file conversion
- Heavy operational effort
- High error risk
- Potential onboarding blocker
After
Automated supported pipelines
- Daily secure collection
- Consistent normalized records
- Manual conversion eliminated
- More client environments supportable
The impact path
Inherited
Prospective clients used reporting systems whose dispute-data structures did not match the platform’s existing ingestion logic or internal models.
Owned
I compared each external schema with the platform’s existing data models and designed the source-to-platform mapping strategy.
Changed
I isolated source-specific ingestion in a standalone scheduled service rather than embedding unreliable external inputs directly in the core application.
Result
Daily — Reliable automated ingestion
What I inherited
The problem and the reality around it
The problem
- Prospective clients used reporting systems whose dispute-data structures did not match the platform’s existing ingestion logic or internal models.
- Incoming files varied across field names, missing values, nested structures, formats, and business meaning.
- Unsupported formats required burdensome, error-prone manual conversion or prevented onboarding entirely.
The constraints
- Daily files arrived through secure SFTP sources and could contain malformed records or unexpected schema changes.
- The existing internal model needed to remain stable rather than expanding for every external field.
- The work had to progress while the six-person team remained responsible for several other active projects.
- Sensitive dispute data, credentials, infrastructure details, and proprietary schemas required careful handling and remain generalized.
What I owned
My responsibility in the work
I compared each external schema with the platform’s existing data models and designed the source-to-platform mapping strategy.
I built most of the standalone Node.js service, including daily SFTP collection, normalization, validation, retry behavior, and API integration.
I coordinated repeated mapping reviews with product and data stakeholders so technical compatibility and business meaning stayed aligned.
I partnered with the DevOps lead on secure initial access and guided a junior product manager through discovery and validation decisions.
I led the production rollout while balancing this work with the team’s other active commitments.
How I approached it
Decisions, tradeoffs, and delivery
I built a standalone Node.js service that collected new files from secure SFTP sources each day, processed them outside the core application, and submitted normalized records through the platform API. I compared each incoming schema with the existing internal models and mapped values that represented the same business concepts, even when their keys or structures differed. When a data point had no equivalent, I evaluated its product and operational importance. High-value information could justify a carefully considered new internal field; low-value unmatched data was excluded rather than adding permanent complexity. The service validated files before submission, flagged failures, retried once, and stopped questionable data from quietly entering production. This separated unreliable external inputs from the main product while creating a controlled path into its existing reporting and dispute workflows.
- I isolated source-specific ingestion in a standalone scheduled service rather than embedding unreliable external inputs directly in the core application.
- I mapped external values to existing internal keys when they represented the same business concept, even when their names and structures differed.
- When no equivalent key existed, I evaluated the data point’s importance and created a new internal field only when the product value justified the added complexity.
- I excluded low-value unmatched fields instead of allowing every external schema to reshape the platform’s internal model.
- I validated files before submission, flagged failures, retried once, and routed the file and warning for investigation when the retry failed.
- I sent normalized records through the platform API so users could work with them through existing dispute management, reconciliation, reporting, filtering, and export workflows.
- I contributed to internal operational tooling that surfaced ingestion activity and failures without exposing proprietary implementation details.
How I led
The team and stakeholder system
The coding effort was concentrated between me and one engineer, with my work focused primarily on the standalone service, SFTP ingestion, schema translation, validation, and API integration. The second engineer contributed primarily to the frontend and internal operational tooling. I reviewed the mappings multiple times with product and data stakeholders because schema translation can be technically valid while still being wrong in business terms. Product helped determine which external values mattered to user workflows, the data team helped validate meaning and treatment, and I partnered with the DevOps lead on the secure initial setup. I was also training a junior product manager and used the project to make the reasoning visible: compare both models, preserve meaning, distinguish important information from noise, and validate assumptions with the people closest to the domain.
Results
What changed
Daily
Reliable automated ingestion
Files were collected, translated, validated, and delivered into the dispute-management platform without manual intervention. · Shared outcome
2–3 months
First pipeline to production
The first new source moved from schema investigation to dependable daily production ingestion within the recalled delivery window. · Shared outcome
Eliminated
Manual conversion for supported pipelines
The automated flow removed the prior burdensome and error-prone conversion step for the reporting systems it supported. · Shared outcome
Leadership evidence
How I moved the people system
- — I connected architecture, data interpretation, product meaning, secure setup, and commercial opportunity into one delivery path.
- — I kept a focused two-engineer implementation moving while leading a six-person team across several concurrent projects.
- — I coached a junior product manager through the reasoning behind mapping, prioritization, and cross-functional validation.
- — I treated schema expansion as a deliberate product decision rather than an automatic response to every unmatched field.
Technical evidence
How I moved the product system
- — Standalone Node.js service for scheduled daily collection from secure SFTP sources.
- — Source-specific translation into a stable internal model with selective schema extension.
- — Pre-ingestion validation, one automatic retry, visible warnings, and controlled exception routing.
- — API-based delivery into existing dispute-management, reconciliation, reporting, filtering, and export workflows.
- — Operational visibility for ingestion activity and failures.
Technical footprint
Technologies and system areas
What I took from it
The lesson I carry forward
The result was more than a parser. Within approximately two to three months, the first new pipeline moved from investigation to reliable daily production ingestion. For the supported pipelines, files processed automatically, records appeared correctly inside the dispute-management platform, and the previous manual conversion step was eliminated. Users could work with the imported data through existing reconciliation, reporting, filtering, export, and case-management capabilities. The internal launch was intentionally quiet, but it expanded what the business could credibly support and allowed the commercial team to approach prospective clients using previously incompatible reporting systems. This project reinforced a principle I still carry: successful integration work translates business meaning—not just field names—while protecting the stability of the system receiving the data.
Related work · Data