NEMS Real-Time Analytics Pipeline
End-to-end data pipeline for NEMS Accounter that cut financial data processing time from one full day to three minutes, enabling real-time decision-making.
Context
NEMS AS built accounting and operational software for the energy sector in Norway. Their flagship product, NEMS Accounter, processed large volumes of financial and operational data from energy installations: metering records, cost allocations, and compliance reporting.
When I joined, the data pipeline ran as a nightly batch job. Reports and analytics were available only after a full day of processing, which meant financial decisions were always made on yesterday's numbers. During my tenure (Feb 2018–Dec 2021), I rebuilt the pipeline from the ground up to run in near-real-time.
What I Built
Real-time data pipeline: Redesigned the core data processing flow from a nightly batch job into an event-driven pipeline. Data flows from source systems into SQS queues, processed by Lambda functions and ECS Fargate tasks in parallel, and materialized into DynamoDB and RDS for querying. Processing that previously took a full day now completes in under three minutes.
Serverless migration: Migrated the job server infrastructure from EC2 instances (always-on, expensive) to a mix of AWS Lambda (short-lived event processors) and ECS Fargate (containerized batch tasks that scale to zero). This reduced infrastructure costs while improving throughput and reliability.
Monitoring dashboards: Built real-time monitoring dashboards that gave operations teams visibility into pipeline health, queue depths, and processing latency. Infrastructure issues that previously surfaced through customer complaints were now detected and resolved proactively.
Excel drag-and-drop import: Developed a Vue.js frontend feature allowing users to import structured data via drag-and-drop Excel uploads, significantly reducing manual data entry time. The backend parsed and validated uploaded files server-side before ingesting them into the pipeline.
Integration-as-a-Service: Implemented connectors for SFTP and Azure Data Lake, enabling NEMS Accounter to ingest data from external sources and partners without custom per-integration engineering work.
Environmental compliance reports: Designed NEMS Permit PPC reporting, generating environmental compliance documentation required by Norwegian regulatory authorities for energy installations.
Outcome
The pipeline rebuild cut processing time from 1 day to 3 minutes, a 99.8% reduction. Financial analysts at client companies could see up-to-date numbers throughout the day instead of waiting overnight. The shift to serverless eliminated the EC2 fleet that ran the batch jobs, reducing infrastructure operating costs and removing a category of scaling bottlenecks.
What I Learned
The hardest part wasn't the technology. It was understanding the domain well enough to decompose the batch job correctly. Energy sector accounting has subtle sequencing requirements: certain aggregations can only run after their dependencies are materialized, and some compliance calculations are stateful across billing periods. Getting the event ordering right required working closely with the domain experts who had built the original batch system. The lesson: before optimizing a pipeline, fully understand why it runs in the order it does.
Key Outcome
Reduced data processing time from 1 day to 3 minutes (99.8% faster), enabling real-time financial analytics for NEMS Accounter.